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Abstract 


Modem  software  development  is  highly  reliant  on  reusable  APIs.  APIs  often  define  usage  protocols  that 
API  clients  must  follow  in  order  for  code  implementing  the  API  to  work  correctly.  Loosely  speaking,  API 
protocols  define  legal  sequences  of  method  calls  on  objects.  In  this  work,  protocols  arc  defined  based  on 
typestates  (Strom  and  Yemini,  1986;  DeLine  and  Fahndrich,  2004b).  Typestates  leverage  the  familial-  intu¬ 
ition  of  abstract  state  machines  to  define  usage  protocols. 

The  goal  of  this  work  is  to  give  developers  comprehensive  help  in  defining  and  following  API  proto¬ 
cols  in  object-oriented  software.  Two  key  technical  contributions  enable  the  proposed  approach:  (1)  Object 
state  spaces  are  defined  with  hierarchical  state  refinements.  Hierarchical  state  spaces  make  specifications 
more  succinct,  elegantly  deal  with  subtyping,  express  uncertainty,  and  enable  more  precise  reasoning  about 
aliasing.  (2)  A  novel  abstraction,  called  access  permissions ,  combines  typestate  and  aliasing  information. 
Access  permissions  capture  developers’  design  intent  regarding  API  protocols  and  enable  sound  modular 
verification  of  API  protocol  compliance  while  allowing  a  great  deal  of  flexibility  in  aliasing  objects. 

This  dissertation  demonstrates  that  typestate-based  protocols  with  state  refinement  and  access  permis¬ 
sions  can  be  used  for  automated,  static,  modular  enforcement  of  API  protocols  in  practical  object-oriented 
software.  Formal  and  empirical  results  show  that  the  presented  approach  captures  common  API  protocols 
succinctly,  allows  sound  modular  checking  of  protocol  compliance  in  object-oriented  code,  can  be  auto¬ 
mated  in  tools  for  mainstream  programming  languages  that  impose  low  annotation  burden  on  developers, 
and  can  check  API  protocols  in  off-the-shelf  software  with  higher  precision  than  previous  approaches. 

This  work  puts  automatic  API  protocol  compliance  checking  within  reach  of  being  used  in  practice.  It 
will  enable  rapid  and  collect  use  of  APIs  during  initial  construction  and  ensure  that  API  clients  and  imple¬ 
mentations  remain  consistent  with  the  specified  protocol  during  maintenance  tasks. 
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Chapter  1 


Introduction 


1.1  APIs  and  Object  Protocols 

Modern  software  development  is  highly  reliant  on  reusable  APIs  (Application  Programming  Interfaces). 
Many  programming  languages  in  industrial  use  such  as  Java  or  C#  include  enormous  “standard”  libraries 
for  everyday  tasks,  and  organizations  often  use  additional  libraries  and  frameworks  that  were  developed 
in-house,  in  the  open-source  community,  or  by  third  parties.  For  example,  the  Java  Database  Connectivity 
API  (JDBC)  defines  canonical  interfaces  for  accessing  relational  databases  from  Java  programs  using  SQL 
queries  which  developers  can  use  without  having  to  worry  about  the  details  of  accessing  the  particular 
database  they  employ. 

Reusable  APIs  often  define  usage  protocols.  Loosely  speaking,  usage  protocols  impose  constraints  on 
the  order  in  which  events  arc  allowed  to  occur.  For  example,  a  database  connection  can  only  be  used  to 
execute  SQL  queries  while  it  is  open.  Running  an  SQL  query  will  return  a  “result  set”  object  that  can  be 
used  to  iterate  over  the  rows  (tuples)  of  the  query  result  using  the  next  operation  (Figure  1.1).  This  operation 
will  return  true  if  the  result  set  was  advanced  to  a  valid  row  and  false  otherwise.  Column  values  can  be 
retrieved  from  valid  rows,  but  attempts  to  retrieve  column  values  in  the  end  state  violate  the  protocol  for 
ResultSet.  Furthermore,  result  sets  can  only  be  used  until  the  connection  they  were  created  on  is  closed.1 

It  has  been  a  long-standing  research  challenge  to  ensure  statically  (before  a  program  ever  runs)  that  API 
protocols  are  followed  in  client  programs  using  an  API.  A  long-overlooked  but  related  problem  is  to  make 
sure  that  the  protocol  being  checked  is  consistent  with  what  the  API  implementation ,  such  as  implementa¬ 
tions  of  JDBC  interfaces  for  a  particular  database,  does.  Both  of  these  challenges  are  complicated  by  object 
aliasing  (objects  being  referenced  and  possibly  updated  from  multiple  places  in  the  program) — the  hallmark 
feature  of  imperative  languages  like  C  and  Java. 

The  goal  of  this  work  is  to  give  developers  comprehensive  help  in  following  API  protocols  as  well  as  to 
allow  them  to  correctly  and  concisely  formalize  protocols  for  their  own  code.  Our  studies  of  APIs,  primarily 
from  the  Java  standard  library,  show  that  challenges  in  this  area  revolve  around  expressiveness,  subtyping, 
inheritance,  and  aliasing.  These  challenges  will  be  described  in  more  detail  below. 

This  dissertation  proposes  a  set  of  techniques  that  address  these  challenges.  The  techniques  can  formal¬ 
ize  and  statically  enforce  API  protocols  in  common  object-oriented  programs.  Case  studies  show  that  these 

’This  description  is  simplified:  connections  and  result  sets  are  related  through  intermediary  “statement”  objects  (Section  7.1.1). 
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CHAPTER  1.  INTRODUCTION 


techniques  can  express  interesting  and  relevant  protocols  of  APIs  defined  in  the  Java  standard  library,  and 
that  the  tools  we  developed  can  automatically  verify  compliance  to  these  protocols  in  open-source  software 
codebases. 

1.1.1  Helping  Developers 

Currently,  it  is  a  painstaking  and  time-consuming  undertaking  for  a  developer  to  gain  familiarity  with  an 
API.  Developers  typically  create  a  series  of  (hopefully  small)  programs  that  exercise  the  API  to  understand 
how  it  works,  for  instance  by  invoking  operations  whose  names  promise  to  be  relevant  for  the  task  at  hand. 
If  the  developer  is  fortunate  then  the  API  implementation  will  throw  meaningful  runtime  exceptions  when 
protocols  are  violated.  But  at  other  times,  API  invocations  may  simply  not  produce  the  expected  result, 
i.e.,  they  silently  fail,  or  protocol  violations  may  corrupt  internal  data  structures  of  the  API  implementa¬ 
tion,  leading  to  subtle  errors  later  (when  these  data  structures  are  accessed  again),  or  even  security  flaws 
exploitable  by  an  attacker.  In  these  cases,  developers  may  be  forced  to  use  the  debugger  to  investigate  the 
API  internals.  If  available,  the  developer  can  also  read  documentation,  example  code,  online  forums,  or  the 
API  implementation  code,  which  may  accelerate  or  decelerate  the  process  depending  on  the  quality  of  these 
arti facts. 2  What  we  need  is  a  situation  where  developers  can  write  code  against  an  unfamiliar  API  and  tools 
will  point  out  when  they  start  violating  protocols,  allowing  developers  to  focus  on  the  task  they  arc  trying  to 
accomplish. 

Even  familial-  API  protocols  can  be  difficult  to  follow  in  all  cases.  Protocols  of  real  APIs  are  complex 
and  easily  violated.  In  particular,  if  objects  created  through  the  API  depend  on  each  other  in  some  way,  are 
stored  in  fields — or  are  otherwise  shared — then  ensuring  compliance  to  API  protocols  becomes  a  non-local 
problem  that  is  challenging  even  for  expert  developers.  Moreover,  if  a  program  behaves  as  expected,  there  is 
no  guarantee  that  API  protocols  will  be  followed  on  all  paths  through  the  program.  This  reduces  developers’ 
confidence  in  their  code,  especially  during  maintenance  tasks,  which  can  lead  to  long  testing  cycles  and 
high  costs  to  fix  bugs  late  in  the  development  process.  We  would  like  to  quickly  check  that  protocols  are 
consistently  followed  when  code  (or  its  protocol)  is  written  or  changed. 

Producing  reusable  APIs  is  time-consuming  as  well  because  protocols  have  to  be  correctly  documented 
and  implementations  have  to  comply  with  their  documented  protocols.  In  order  to  avoid  erratic  behavior  at 
runtime,  API  implementations  have  to  protect  themselves  against  protocol  violations  by  clients,  requiring 
consistent  and  expensive  runtime  checks  on  method  arguments.  If  data  structures  are  shared  between  API 
clients  and  implementation  then  API  implementation  code  has  to  be  prepared  for  client  modifications  of 
these  data  structures.  Reentrancy  and  potential  overriding  further  exacerbate  the  challenges  for  API  devel¬ 
opers.  Here,  we  would  like  to  check  that  “bad  states”  are  avoided  and  implementations  are  consistent  with 
documented  protocols.  Moreover,  if  clients  are  trustworthy,  we  could  potentially  speed  up  programs  by 
removing  defensive  runtime  checks. 


Using  this  work.  For  the  purpose  of  this  work,  we  roughly  distinguish  API  designers,  API  client  devel¬ 
opers,  and  API  implementers.  Reflecting  upon  the  scenarios  described  above,  our  goals  include  aiding  API 
designers  in  documenting  API  protocols  and  client  developers  in  understanding  APIs;  to  assure  to  client 

2The  author  had  an  experience  with  the  Eclipse  IDE's  “builder”  infrastructure  where  (a)  names  used  in  the  API  did  not  reflect 
their  meaning,  (b)  the  documentation  did  not  state  important  rules,  and  (c)  sample  code  included  with  Eclipse  was  faulty  (fixed 
upon  request  by  the  author  with  bug  263807).  Several  hours  of  using  the  debugger  finally  shed  light  on  the  problem. 
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nextQ  /  true  next()  /  false 
(a)  Protocol  visualized  similar  to  a  Statechart  using  state  refinement 


Figure  1.1:  Simplified  JDBC  result  set  protocol.  Return  values  for  next  are  included  to  show  the  possible 
outcomes  of  dynamic  state  tests. 


developers  that  API  protocols  are  followed  in  their  code;  and  to  help  API  implemented  create  dependable 
API  implementations.  In  order  to  be  applicable  to  large  codebases,  we  would  like  to,  as  much  as  possible, 
check  protocols  automatically  (instead  of  relying  on  manual  proofs)  and  in  a  way  suitable  for  interactive  use 
(in  order  to  support  developers  while  they  write  or  maintain  code).  The  work  presented  in  this  dissertation 
accomplishes  these  goals  as  follows: 

•  Documentation  and  understanding.  API  designers  can  use  hierarchical  state  machines,  such  as  the 
one  illustrated  in  Figure  1.1(a),  to  succinctly  describe  API  protocols.  Additionally,  they  use  “access 
permissions”  to  describe  how  API  methods  will  access  and  use  method  parameters.  Hierarchical  state 
machines  and  access  permissions  also  address  a  number  of  other  concerns  and  are  described  in  more 
detail  below. 

Figure  6.1  shows  a  description  of  the  ResultSet  protocol  discussed  in  this  chapter  using  the  anno¬ 
tations  defined  by  our  tool,  Plural.  There,  Java  annotations  ©Full  and  ©Pure  indicate  modifying  and 
reading  access  to  the  receiver  object,  respectively,  and  specify  states  required  or  ensured  by  the  an¬ 
notated  methods,  directly  encoding  the  protocol  shown  in  Figure  1.1(a).  The  conventional  Java  type 
signature  for  ResultSet  is  unchanged  by  these  annotations. 

These  annotations  allow  API  designers  to  concisely  and  formally  document  then  protocols;  they  can 
also  help  developers  using  an  API  understand  how  the  API  works,  even  without  tool  involvement. 

•  Automation,  assurance,  and  interactivity.  Our  tool.  Plural,  automatically  checks  at  compile-time 
whether  a  given  API  client  program  complies  with  API  protocols  described  as  above.  Plural  works 
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with  conventional  Java  code,  and  similar  tools  could  be  developed  for  other  programming  languages. 
Plural  determines  for  every  variable  at  every  program  point  what  is  known  about  the  state  of  the 
referenced  object.  This  information  is  known  as  the  object’s  current  “typestate”  (Strom  and  Yemini, 
1986)  and  allows  assuring  that  API  protocols  arc  followed. 

Plural  may  need  additional  annotations  for  method  parameters  and  instance  fields  to  check  individual 
methods  in  a  API  client  program  independently  from  the  rest  of  the  program.  Our  case  studies  show 
that  this  approach  requires  only  about  one  developer-provided  annotation  per  method  in  an  API  client 
program.  These  additional  annotations  enable  quick  checking  of  individual  methods  (currently  around 
100ms)  and  allows  making  assumptions  about  unfinished  or  unavailable  parts  of  the  program. 

API  client  developers  can  use  Plural  interactively  while  they  write  code  in  their  development  envi¬ 
ronments.  Figure  6.2  shows  a  small  Re  suit  Set  client  with  an  annotation  for  a  method  parameter. 
This  annotation  allows  making  all  calls  on  the  annotated  parameter,  rs ,  that  arc  legal  according  to  its 
protocol  (Figure  6.1).  Plural  also  detects  a  protocol  violation.  When  the  developer  fixes  the  problem 
she  can  re-run  Plural  to  assure  that  the  modified  code  respects  all  API  protocols. 

•  Dependability.  Separately,  developers  of  API  implementations  can  use  Plural  to  make  sure  that  their 
code  is  consistent  with  the  published  protocol  (see  Section  3.2). 

This  work  takes  the  point  of  view  that  a  protocol  is  paid  of  an  APFs  interface  that  should  be  captured 
as  paid  of  that  interface  and  subsequently  can  be  enforced  (Section  6.6  summarizes  specific  use  cases  of  the 
tool).  This  improves  encapsulation  of  APIs  because  protocols  can  be  captured  abstractly  without  specific 
knowledge  of  implementation  details.  It  also  improves  their  reusability  because  API  clients  can  be  assured 
that  they  use  the  API  according  to  its  protocol. 

1.1.2  Challenges 

The  approach  presented  in  this  dissertation  is  based  on  existing  work  on  typestates  (Strom  and  Yemini, 
1986).  The  idea  behind  typestates  is  to  express  protocols,  like  the  ResultSet  protocol  discussed  above,  as 
state  machines  (Figure  1.1).  Such  protocols  allow  tracking  (statically  or  dynamically)  the  current  state  of 
each  object.  As  such,  typestates  are  an  intuitive  mechanism  for  describing  protocols  which  can  be  enforced 
automatically  in  a  sound  and  modular  fashion  (DeLine  and  Fahndrich,  2001,  2004b).  Typestates  arc  par¬ 
ticularly  appealing  for  object-oriented  software  because  typestates  characterize  objects  abstractly,  without 
revealing  implementation  details  (DeLine  and  Fahndrich,  2004b).  Subtyping,  however,  creates  challenges 
for  typestate-based  approaches. 

For  this  thesis  we  studied  five  commonly  used  APIs  from  the  Java  standard  library  including  the  Java 
Collections  and  Database  Connectivity  (JDBC)  APIs  (Bierhoff  and  Aldrich,  2005,  2007b;  Bierhoff  et  ah, 
2009). 3  This  section  summarizes  the  challenges  we  found  with  expressing  and  enforcing  protocols  for  these 
APIs  using  typestates,  including  expressiveness,  subtyping  and  inheritance,  and  aliasing.  The  next  section 
will  provide  an  overview  of  how  these  challenges  arc  addressed  in  this  work. 

Expressiveness.  Real  APIs  define  complex  protocols,  and  it  is  crucial  that  developers  be  able  to  express 
their  protocols;  otherwise,  we  cannot  check  code  for  compliance  to  these  protocols.  As  discussed  in  more 

3Four  of  them  are  subject  of  case  studies  presented  in  Section  7.1. 
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detail  in  Section  7.4.1,  we  found  three  recurring  patterns  (which  appeared  in  at  least  3  of  the  5  APIs  we 
studied)  of  protocols  which  arc  not  sufficiently  expressible  in  previous  protocol  enforcement  approaches. 
One  of  them,  dependent  objects,  is  not  readily  checkable  with  any  automated  modular  program  verification 
approach  (such  as  Flanagan  et  al.,  2002;  Barnett  et  ah,  2004)  of  which  we  arc  aware.  For  example,  dependent 
objects  can  be  found  in  the  JDBC  API:  multiple  queries  can  be  run  on  a  single  connection  to  the  database, 
resulting  in  multiple  result  sets.  While  result  sets  arc  in  use,  they  require  the  connection  to  remain  open,  i.e., 
ResultSet  objects  depend  on  the  Connection  object  they  were  created  on. 

Moreover,  flat  state  machines  quickly  suffer  from  state  explosion  problems  when  trying  to  describe 
the  intricacies  of  API  protocols,  making  them  tedious  to  manually  define  and  understand  (Bierhoff  and 
Aldrich,  2005).  For  example,  we  found  52  distinct  states  when  specifying  the  Java  standard  library  class 
PipedlnputStream  (Bierhoff  and  Aldrich,  2005).  To  be  comprehensible,  specifications  of  such  protocols 
should  remain  succinct,  a  common  challenge  with  approaches  based  on  state  machines  that  is  not  addressed 
in  previous  typestate-based  techniques. 


Subtyping  and  inheritance.  Subtyping  and  inheritance  arc  hallmark  features  of  object-oriented  program¬ 
ming.  Subtypes  should  respect  behavioral  subtyping  (Liskov  and  Wing,  1994)  and  remain  substitutable  for 
supertypes.  However,  typestates  have  traditionally  impeded  substitutability  (Bierhoff  and  Aldrich,  2005): 
subtypes  often  need  additional  states  to  describe  their  protocols,  which  prohibits  objects  from  being  substi¬ 
tuted  when  they  arc  in  states  unknown  in  supertypes.  For  instance,  we  distinguish  open  and  closed  states  for 
Connection  objects,  but  not  for  other  types  of  objects.  Previous  approaches  would  forbid  substituting  an 
open  Connection  for  an  Object  (its  supertype)  because  Object  only  defines  a  “default”  state  that  is  not 
the  same  as  open  (DeLine  and  Fahndrich,  2004b). 

Behavioral  subtyping  requires  protocol  inheritance:  protocols  defined  in  supertypes  have  to  be  respected 
when  using  subtypes.  Subtypes  may,  however,  want  to  concretize  an  inherited  imprecise  protocol  or  react  to 
new  input,  which  is  not  allowed  in  previous  typestate-based  approaches  (Bierhoff  and  Aldrich,  2005). 

Finally,  when  subclasses  override  a  method  in  previous  approaches,  they  cannot  choose  whether  they 
want  to  invoke  the  overridden  method  or  not — that  choice  is  made  by  the  superclass  (DeLine  and  Fah¬ 
ndrich,  2004b).  This  limits  opportunities  for  reuse  and  creates  problems  for  subclasses  such  as  Java’s 
Buf  f  eredlnputStream  which  only  sometimes  invokes  overridden  methods  (details  in  Section  3.2.2). 


Automated  sound  and  modular  protocol  checking  under  aliasing.  Most  software  written  in  imperative 
and  object-oriented  programming  languages  crucially  relies  on  aliasing.  Aliasing  means  that  a  single  object 
(or  memory  location)  is  referenced  from  multiple  places  in  the  program.  We  will  often  say  that  these 
references  “alias”  the  commonly  referenced  object. 

Aliasing  greatly  complicates  the  ability  of  developers  and  tools  alike  to  ensure  statically  that  a  program 
module  complies  with  API  protocols.  This  is  because  aliasing  makes  reasoning  about  program  behavior  a 
non-local  problem:  aliasing  is  typically  used  to  give  different  parts  of  a  program  access  to  the  same  object. 

It  turns  out  that  aliasing  is  not  only  common  in  imperative  programs,  but  is  also  inherent  to  many  APIs. 
For  instance,  the  dependent  objects  pattern  mentioned  above  can  be  seen  as  a  particular  form  of  aliasing 
where  result  sets  alias  the  connection  they  were  created  on.  However,  the  connection  will  typically  also  be 
aliased  from  the  client  program  that  created  connections  and  result  sets.  The  trouble  now  is  to  ensure  that 
the  client  program  closes  the  connection  only  after  closing  all  the  result  sets  that  depend  on  it. 
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Aliasing  also  makes  it  more  difficult  to  check  whether  a  class  implements  its  specified  protocol  because 
reentrant  callbacks  through  aliases  can  lead  to  unexpected  state  changes.  For  instance,  we  can  trigger  the 
equivalent  of  a  buffer  overrun  in  Java’s  Buf  f  eredlnputStream  through  reentrant  callbacks  (Bierhoff  and 
Aldrich,  2007a).  By  reentrancy  we  mean  that  a  method  executes  within  the  dynamic  scope  of  the  same  or 
another  method  defined  in  the  same  class  and  with  the  same  receiver  object. 

Existing  automated  static  protocol  checking  approaches  fall  into  two  categories.  Some  protocol  checkers 
arc  whole -program  (global)  analyses,  i.e.,  they  check  an  entire  codebase  at  once.  Whole-program  analyses 
typically  account  for  aliasing,  but  they  can  be  imprecise,  they  cannot  analyze  incomplete  programs,  and  they 
arc  typically  too  slow  to  be  used  interactively  by  a  developer. 

The  other  category  of  protocol  checkers  is  modular.  Modular  protocol  checkers  can  support  developers 
while  they  write  code:  like  a  typechecker,  they  check  each  method  separately  for  protocol  violations  while 
assuming  the  rest  of  the  system  to  behave  as  specified.  The  trade-off  has  been  that  modular  checkers  require 
code  to  follow  pre-defined  patterns  of  aliasing.  Generally  speaking,  state  changes  arc  only  allowed  where 
the  checker  is  sure  that  the  changing  object  is  not  accessed  through  other  references,  simplifying  sound 
reasoning  about  the  states  of  referenced  objects. 

This  approach  of  largely  forbidding  aliasing  has  serious  drawbacks.  First,  many  examples  of  realistic 
code  arc  excluded.  Moreover,  from  a  developer’s  point  of  view,  the  boundaries  of  what  a  checker  supports 
might  not  fit  with  the  best  implementation  strategy  for  a  particular-  problem.  Finally,  aliasing  restrictions 
arguably  leave  developers  alone  just  when  they  have  the  most  trouble  in  reasoning  about  their  code,  namely, 
in  the  presence  of  subtle  aliasing.  This  is  exactly  the  case  for  JDBC:  aliasing  restrictions  prevent  existing 
modular  checking  approaches  from  handling  multiple  result  sets  over  a  single  connection,  although  using 
JDBC  is  tricky  precisely  because  of  possible  interference  through  aliases. 

Unfortunately,  it  is  difficult  to  offer  programmers  the  flexibility  to  alias  objects  in  their  code  while 
maintaining  precise  sound  and  modular  reasoning  about  protocol  compliance.  Soundness  means  that  we 
will  not  miss  potential  protocol  violations  in  any  execution  of  the  program.4  With  modularity  we  mean  that 
we  can  analyze  a  part  of  the  program  (a  single  method  in  our  implementation)  for  protocol  compliance. 
Finally,  precision  means  keeping  the  number  of  false  positives  small.  (False  positives  are  protocol  violation 
warnings  that  do  not  result  in  actual  protocol  violations  at  run  time.)  Since  we  are  dealing  with  an  in  general 
statically  undecidable  problem,  avoiding  both  false  negatives  and  false  positives  is  impossible,  but  a  precise 
sound  analysis  avoids  false  negatives  completely  and  keeps  false  positives  to  a  minimum. 

Unrestricted  aliasing  defies  precise,  sound,  modular  reasoning  because  the  presence  of  other  references 
to  objects  manipulated  in  the  current  module  allows  the  possibility  of  these  objects  changing  state  at  virtually 
any  moment  while  code  from  the  current  module  executes,  even  when  the  program  runs  sequentially  in  a 
single  thread.  In  particular-,  when  analyzing  the  code  in  a  given  method,  objects  could  change  state  as  a 
result  of  every  method  call  in  the  code.  Soundness  would  require  “forgetting”  the  state  of  all  objects  after 
every  method  call,  which  would  lead  to  a  very  imprecise  analysis.  Not  forgetting  the  state  of  potentially 
aliased  objects  will  result  in  a  more  precise,  but  grossly  unsound,  analysis.  Achieving  both  soundness  and 
precision  in  a  modular  analysis  is  exceedingly  difficult  in  the  presence  of  aliasing. 


4Errors  related  to  race  conditions  are  beyond  the  scope  of  this  dissertation.  However,  this  work  can  be  soundly  extended  to 
concurrent  programs  (Beckman  et  al.,  2008). 
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1.2  This  Dissertation 

The  research  presented  in  this  dissertation  addresses  the  challenges  we  outlined  above.  There  arc  two  key 
technical  innovations  enabling  the  work  presented  in  this  dissertation,  state  refinement  and  access  permis¬ 
sions.  We  briefly  introduce  them  below  before  we  discuss  how  they  help  dealing  with  API  protocols. 

•  State  refinement.  Instead  of  using  a  flat  state  space  with  a  set  of  mutually  exclusive  states,  in  this 
work,  state  spaces  arc  hierarchical.  Sets  of  mutually  exclusive  states  refine  more  coarse-grained  states, 
possibly  along  different  orthogonal  dimensions,  making  protocol  specifications  akin  to  Statecharts 
(Harel,  1987).  This  concept  makes  specifications  more  succinct,  elegantly  deals  with  subtyping,  and 
enables  more -precise  reasoning  about  aliasing.  Additionally,  state  refinement  can  be  used  to  encode 
uncertainty  (for  instance,  which  state  a  result  set  is  in  after  a  call  to  next),  which  is  useful  in  cases 
where  APIs  make  internal  choices. 

•  Access  permissions  are  an  abstraction  that  combines  typestate  with  aliasing  information  about  ob¬ 
jects  (Bierhoff  and  Aldrich,  2007b).  Developers  use  access  permissions  to  express  the  design  intent 
of  their  protocols  in  annotations  on  methods  and  classes.  For  every  reference,  access  permissions 
flexibly  delineate  the  effects  of  aliasing,  allowing  protocol  checking  to  be  appropriately  conservative 
in  reasoning  about  that  reference.  The  primary  mechanisms  for  constraining  aliasing  effects  arc  to 
restrict  aliases  to  be  read-only,  i.e.,  to  not  modify  state,  and  to  make  references  respect  a  state  guar¬ 
antee,  i.e.,  to  not  leave  a  certain  state.  These  mechanisms  achieve  high  precision  in  tracking  effects  of 
possible  aliases. 

The  following  subsections  discuss  in  detail  how  state  refinement  and  access  permissions  address  the 
challenges  discussed  above  (expressiveness,  subtyping  and  inheritance,  and  aliasing).  Afterwards  we  sum¬ 
marize  the  contributions  of  this  dissertation.  Section  3  will  discuss  the  approach  in  detail. 

1.2.1  Expressiveness 

Expressive  specifications 

It  is  crucial  that  developers  be  able  to  express  their  protocols.  Our  approach  is  able  to  capture  challenging 
API  protocol  patterns  in  common  use.  In  particular,  dependencies  between  objects  can  be  expressed  with 
permissions,  and  uncertainty  can  be  captured  by  referring  to  refined  states.  In  addition,  we  employ  a  logic 
for  expressing  complex  protocols  with  disjunctions  and  implications.  In  practice,  we  only  needed  this  power 
in  very  stylized  ways,  namely  for  “dynamic  state  tests”  (such  as  the  next  method  in  Figure  1.1,  whose  return 
value  “indicates”  or  implies  that  the  result  set  is  in  a  certain  state),  “method  cases”  (which  can  be  seen  as 
conjunctions  between  function  types,  see  Dunfield  and  Pfenning  (2004)),  and  for  recovering  access  to  fields 
of  dead  objects  (which  we  model  with  a  linear  implication). 

Succinct  specifications 

Protocols  of  realistic  interfaces  quickly  become  complex  (Bierhoff  and  Aldrich,  2005;  Bierhoff,  2006).  To 
be  comprehensible,  specifications  of  such  protocols  should  remain  succinct. 

In  our  approach,  “new”  typestates  are  introduced  by  refining  an  existing  one  into  a  set  of  more  fine¬ 
grained  states.  This  idea  corresponds  to  OR-states  in  Statecharts  (Harel,  1987). 
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State  refinement  allows  more-succinct  specifications  in  many  cases.  For  instance,  it  allows  the  ResultSet 
protocol  from  the  previous  section  to  be  specified  with  5  state  transitions.  The  same  protocol  requires  12 
transitions  in  a  flat  state  machine  (Figure  1.1). 

By  allowing  states  to  be  refined  multiple  times  the  proposed  approach  avoids  state  explosion  in  many 
cases.  We  allow  multiple  orthogonal  refinements  of  the  same  typestate.  Such  orthogonal  state  dimensions 
correspond  to  AND-states  in  Statecharts  (which  arc  used  to  specify  parallel  state  machines)  and  reduce  the 
number  of  distinguished  states  (Bierhoff  and  Aldrich,  2005).  Likewise,  the  overhead  for  specifying  methods 
can  be  further  reduced  because  specifications  only  need  to  concern  themselves  with  state  dimensions  relevant 
to  the  method’s  behavior  (see  Section  3.1). 

1.2.2  Subtyping  and  Inheritance 
Subtype  substitutability 

Subtyping  and  inheritance  arc  hallmark  features  of  object-oriented  programming.  Subtyping  requires  objects 
of  subtypes  to  be  substitutable  for  supertypes.  State  refinement  as  introduced  above  helps  with  subtype 
substitutability:  typestates  defined  in  supertypes  arc  inherited  and  can  be  refined  by  subtypes.  Unlike  with 
existing  approaches,  this  guarantees  substitutability  because  every  typestate  of  a  subtype  has  a  (possibly  less 
precise)  corresponding  typestate  in  its  supertypes. 


Behavioral  subtyping 

It  is  a  common  desideratum  in  behavioral  specifications  that  subtypes  be  able  to  change  protocols  defined 
in  supertypes.  In  particular,  a  subtype  might  want  to  define  a  more -precise  protocol  than  required  by  its 
supertypes,  or  react  to  additional  inputs.  Our  approach  allows  such  protocol  changes  as  long  as  behavioral 
subtyping  (Liskov  and  Wing,  1994)  is  preserved,  i.e.,  the  new  protocol  is  guaranteed  to  be  compatible  with 
the  original  protocol.  Behavioral  subtyping  is  ensured  by  checking  that  specifications  of  overriding  methods 
logically  imply  specifications  of  overridden  methods. 


Flexible  overriding 

Inheritance  is  handled  in  a  novel  way,  giving  subclasses  flexibility  in  method  overriding.  Subclasses  can 
choose  whether  and  when  they  want  to  call  overridden  methods.  This  is  necessary  for  reasoning  about 
realistic  examples  of  inheritance  such  as  Java’s  Buff  eredlnputStream  (details  in  Section  3.2.2). 

1.2.3  Aliasing  Flexibility 

Aliasing  is  a  significant  complication  in  checking  whether  clients  observe  a  protocol:  a  client  does  not 
necessarily  know  whether  a  reference  to  an  object  is  the  only  reference  that  is  active  at  a  particular  execution 
point.  This  also  makes  it  more  difficult  to  check  whether  a  class  implements  its  specified  protocol  because 
reentrant  callbacks  through  aliases  can  again  lead  to  unexpected  state  changes. 

This  dissertation  contributes  a  sound  modular  typestate  verification  approach  that  allows  a  great  deal 
of  flexibility  in  aliasing.  For  each  reference,  it  tracks  the  extent  of  possible  aliasing,  and  is  appropriately 
conservative  in  reasoning  about  that  reference.  This  helps  developers  account  for  object  manipulations  that 
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may  occur  through  aliases.  High  precision  in  tracking  effects  of  possible  aliases  together  with  systematic 
support  for  “dynamic  state  tests”,  i.e.,  runtime  tests  on  the  state  of  objects,  make  this  approach  feasible. 

Precise  alias  tracking  is  achieved  with  a  novel  abstraction,  access  permissions,  that  combines  typestate 
with  aliasing  information  about  objects.  Developers  use  access  permissions  to  express  their  protocols  in 
annotations  on  methods  and  classes.  A  modular  checking  approach  verifies  that  code  follows  declared 
protocols  (both  on  the  client  and  provider  side  of  an  API). 

Access  permissions  are  associated  with  object  references  (“pointers”)  and  govern  access  to  the  refer¬ 
enced  object.  We  distinguish  exclusive  (unique),  exclusive  modifying  (full),  read-only  (pure),  immutable, 
and  shared  access  (Table  1.1).  Furthermore,  permissions  include  a  state  guarantee,  a  state  that  the  method 
promises  not  to  leave  (Bierhoff  and  Aldrich,  2007b).  For  example,  next  can  promise  not  to  leave  the  open 
state  (Figure  1.1).  Reasoning  with  permissions  has  the  flavor  of  rely-guarantee  reasoning  (Giannakopoulou 
et  ah,  2004)  because  permissions  allow  making  assumptions  about  the  rest  of  the  program  which  are  then 
validated  by  putting  restrictions  on  other  permissions. 

Permissions  capture  three  kinds  of  information: 

1.  What  kinds  of  references  exist?  We  distinguish  read-only  and  modifying  access  separately  through 
the  current  and  all  other  references  to  the  same  object,  leading  to  the  five  kinds  of  permissions  shown 
in  Table  1.1. 

2.  What  state  is  guaranteed?  References  can  rely  on  the  guaranteed  state  even  if  the  referenced  object 
could  be  modified  by  other  references. 

3.  What  is  known  about  the  current  state  of  the  object?  In  order  to  enforce  protocols,  we  need  to  keep 
track  of  what  state(s)  the  referenced  object  is  currently  in.  Knowledge  about  the  current  state  at  least 
includes  the  guaranteed  state  but  can  be  more  specific. 

Permissions  can  only  co-exist  if  they  do  not  violate  each  other’s  assumptions.  Thus,  the  following 
aliasing  situations  can  occur  for  a  given  object:  a  single  reference  (unique),  a  distinguished  writer  reference 
(full)  with  many  readers  (pure),  many  writers  (share)  and  many  readers  (pure),  and  no  writers  and  only 
readers  (immutable  and  pure). 

Permissions  are  linear — i.e.,  they  cannot  be  duplicated — in  order  to  preserve  consistency  between  per¬ 
missions  for  the  same  object.  But  unlike  linear  type  systems  (Wadler,  1990),  they  allow  aliasing.  This  is 
because  permissions  can  be  split  when  aliases  arc  introduced.  For  example,  we  can  split  a  unique  permission 
into  a  full  and  a  pure  permission,  written  unique  ^  full  ®  pure,  to  introduce  a  read-only  alias.  Using  frac¬ 
tions  (Boyland,  2003)  we  can  also  merge  previously  split  permissions  when  aliases  disappear  (e.g.,  when  a 
method  returns).  This  supports  recovering  a  more  powerful  permission.  Fractions  arc  rational  numbers  be¬ 
tween  0  and  1  expressing  how  often  apermission  was  split.  For  example,  full  ^  \  -share®  |  -share  ^  full.5 
Splitting  a  permission  into  two  means  replacing  it  with  two  new  permissions  whose  fractions  sum  up  to  the 
fractions  in  the  permission  being  replaced.  Merging  two  permissions  does  the  opposite. 

Access  permissions  offer  programmers  flexibility  to  alias  objects  in  their  code  while  maintaining  precise 
sound  and  modular  reasoning  about  protocol  compliance  when  these  objects  are  used.  The  ability  to  analyze 
programs  in  a  modular  way  is  crucial  for  at  least  four  reasons: 

3In  previous  work,  fractions  below  one  make  objects  immutable  (Boyland,  2003);  in  our  approach,  they  can  alternatively  indicate 
shared  modifying  access,  as  in  this  example. 
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Access  through 
other  permissions 
None 
Read-only 
Read/write 


Current  permission  has  . . . 

Read/write  access  Read-only  access 

unique  (Boyland,  2003)  - 

full  (Bierhoff,  2006)  immutable  (Boyland,  2003) 

share  (DeLine  and  Fahndrich,  2004b)  pure  (Bierhoff,  2006) 


Table  1.1:  Access  permission  taxonomy 


1 .  It  simplifies  scaling  to  large  programs  because  the  program  can  be  analyzed  in  small  chunks. 

2.  It  enables  analyzing  a  program  module  independently  from  its  clients  and  independently  from  the 
libraries  and  frameworks  the  module  relies  upon.  This  allows  our  approach  to  check  compliance  of 
API  implementations  to  their  declared  protocols.  In  contrast,  whole-program  analyses  cannot  check 
an  API  implementation  independently  from  its  clients. 

3.  It  also  is  crucial  for  using  our  approach  in  practical  settings,  where  different  modules  of  a  program 
arc  provided  by  different  groups  of  developers.  In  many  cases,  developers  may  not  even  have  access 
to  all  the  source  code  in  a  project.  In  these  situations,  our  approach  can  check  protocol  compliance  in 
one  module  independently  from  the  others. 

4.  Modularity  makes  our  approach  akin  to  a  conventional  typechecker,  allowing  developers  to  check 
protocols  (automatically)  while  they  perform  development  and  maintenance  tasks. 

As  detailed  in  the  previous  section,  unrestricted  aliasing  defies  precise,  sound,  modular  reasoning  be¬ 
cause  the  presence  of  other  references  would  require  “forgetting”  the  state  of  all  objects  after  every  method 
call,  which  would  lead  to  a  very  imprecise  analysis.  Permissions  obviate  or  limit  the  need  to  forget  states, 
leading  to  high  reasoning  precision  in  many  cases.  Therefore,  our  approach  balances  aliasing  restrictions 
with  reasoning  precision  in  a  novel  way  that  permits  aliasing  patterns  which  could  not  be  analyzed  precisely 
in  a  modular  fashion  before. 

1.2.4  Contributions 

This  dissertation  demonstrates  that  state  refinement  and  access  permissions  can  express  interesting  protocols 
of  real  object-oriented  APIs  and  can  enforce  these  protocols  in  practical  object-oriented  code.  Contributions 
of  this  dissertation  include  the  following: 

1.  The  approach  improves  the  expressiveness  of  typestate-based  protocol  specifications  for  capturing 
protocols  commonly  occurring  in  practice.  For  example,  in  the  result  set  protocol  discussed  above,  the 
next  method  can  transition  to  two  different  states  (Figure  1.1).  This  internal  choice  can  be  captured 
with  this  approach  but  not  with  existing  typestate-based  approaches.  Our  approach  can  also  capture 
dependent  object  patterns,  which  are  insufficiently  supported  in  existing  modular  program  verification 
approaches  (see  below). 
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2.  The  approach  makes  typestates  interact  more  seamlessly  with  subtyping  and  inheritance  than  previous 
typestate -based  and  other  static  protocol  enforcement  techniques.  In  particular,  subtypes  arc  guaran¬ 
teed  to  be  substitutable  and  can  define  new  protocols,  as  long  as  they  remain  compatible  with  protocols 
defined  in  supertypes,  and  subc/rm^s  are  free  to  use  or  not  use  code  inherited  from  superclasses. 

3.  The  approach  offers  flexibility  in  dealing  with  aliasing,  i.e.,  the  presence  of  multiple  references  to  the 
same  object,  while  maintaining  modularity  and  soundness  in  protocol  checking.  Previous  modular 
protocol  checkers  only  allow  state  changes  when  all  references  to  an  object  arc  known;  when  a  checker 
loses  track  of  a  reference  (sometimes  called  an  “escaping  alias”)  then  no  further  state  changes  arc 
allowed.  The  approach  presented  in  this  dissertation  supports  such  escaping  aliases  while  maintaining 
modularity  in  checking.  This  enables  protocol  enforcement  in  many  examples  of  API  protocols, 
including  result  sets  over  database  connections,  that  could  not  be  verified  in  a  modular-  fashion  before. 

4.  The  presented  protocol  checking  approach  can  be  automated  in  practical  tools  for  conventional  pro¬ 
gramming  languages  and  therefore  is  available  directly  to  developers.  Common  API  protocol  patterns 
can  be  supported  with  such  tools.  We  provide  a  prototype  tool.  Plural,  that  checks  API  protocols 
in  Java.  It  is  based  on  a  inference  system  for  polymorphic  fractional  permissions,  i.e.,  it  supports 
quantification  over  exactly  how  often  permissions  were  split  before  reaching  a  given  program  point, 
facilitating  modular  protocol  specifications. 

5.  Applicability  to  practical  software.  Case  studies  show  that  the  approach  can  express  interesting  API 
protocols  occurring  in  practice  with  moderate  overhead.  On  average,  we  needed  about  2  annota¬ 
tions  per  API  method  to  define  protocols.  Case  studies  with  open-source  programs  show  that  we 
can  also  check  compliance  to  these  protocols  with  sufficient  precision  (less  than  1  false  positive  in 
100  LOC),  low  annotation  overhead  (about  1  annotation  per  method),  and  sufficient  performance  for 
interactive  use  (about  100ms  per  method  on  average).  One  of  our  case  studies  suggests  that  our  ap¬ 
proach  can  check  protocol  compliance  in  many  situations  that  cause  imprecisions  in  state-of-the-art 
whole-program  protocol  analyses  (Naeem  and  Lhotak,  2008;  Bodden  et  ah,  2008). 

While  aliasing  flexibility  is  important  for  many  reasons,  our  approach  enables  expressing  a  common  API 
protocol  pattern  that  to  our  knowledge  is  not  readily  expressed  in  any  previous  modular  program  verification 
approach:  dependent  objects.  This  pattern  describes  situations  in  which  several  objects  depend  on  another 
object.  Sometimes  it  is  just  one  object  that  depends  on  another,  such  as  a  buffered  “wrapper”  stream  around 
another  stream  (cf.  Section  3.2),  in  which  case  one  could  use  ownership  (e.g.  Barnett  et  al.,  2004)  to  model 
such  dependencies.  But  ownership  does  not  help  when  multiple  objects  depend  on  a  single  “server”  object. 
For  example,  all  result  sets  depend  on  the  connection  they  were  created  on  to  remain  open  as  long  as 
result  sets  are  used.  Notice  that  we  do  not  have  one  result  set  depend  on  “its”  connection,  which  would  be 
expressible  with  ownership,  but  many. 

We  can  model  these  dependencies  as  aliasing  inherent  in  the  API :  An  object  is  aliased  from  multiple 
other  objects,  all  of  which  are  defined  in  the  API.  As  an  added  benefit,  we  can  allow  objects  to  be  aliased 
both  from  other  objects  defined  in  the  API  and  by  API  clients.  For  example,  with  permissions,  clients  can 
use  statements  to  run  SQL  queries  and  also  access  the  connection  directly — as  long  as  they  do  not  close 
it.  Naeem  and  Lhotak  (2008)  have  pointed  out  that  existing  typestate -based  approaches  (including,  in  their 
assessment,  this  work)  cannot  express  such  protocols.  In  fact,  no  existing  automated  modular  program 
verification  approach  that  we  are  aware  of  can  readily  express  these  protocols.  As  Chapter  7  shows,  we  have 
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been  successful  in  encoding  and  enforcing  the  dependencies  we  found  (objects  remaining  in  a  given  state 
and  objects  being  immutable)  with  access  permissions. 


1.3  Potential  Impact 

This  work  puts  API  protocol  enforcement  within  reach  of  software  engineering  practice.  The  work  allows 
building  automated  tools,  such  as  the  one  described  in  this  dissertation,  for  helping  developers  deal  with 
real  API  protocols  in  practical  software.  This  work  enables  rapidly  and  correctly  using  APIs  during  both 
software  construction  and  maintenance.  APIs  will  become  more  dependable  as  a  result  of  checking  their 
implementations  against  their  declared  protocols.  Finally,  evolution  of  API  clients  and  implementations 
during  maintenance  tasks  is  simplified. 

Furthermore,  the  intuition  of  state  machines  is  widely  used  to  specify  behavior,  and  notably  in  specifying 
embedded  systems.  Therefore,  this  work  might  be  another  step  along  the  path  of  establishing  a  rigorous 
connection  between  systems  specifications  and  implementations. 

The  author  dares  to  speculate  that  a  time  of  Pluralism — “Permissions  Let  Us  Reason  about  ALiasing 
In  a  Static  Manner” — is  upon  us.  Fractional  permissions  have  already  received  a  great  deal  of  attention 
in  the  context  of  enforcing  race  freedom  (Boyland,  2003;  Terauchi  and  Aiken,  2008)  and  separation  logic 
(Bornat  et  ah,  2005).  This  dissertation  proposes  a  broader  set  of  permissions  for  fine-grained  aliasing  con¬ 
trol  and  shows  that  permissions  allow  precise,  sound,  and  modular  reasoning  about  program  behavior  while 
providing  aliasing  flexibility.  Furthermore,  permissions  can  express  common  aliasing  patterns  that  cannot 
be  captured  with  state-of-the-art  automated  program  verification  techniques,  which  arc  based  on  ownership 
(Bierhoff  and  Aldrich,  2008),  and  could  therefore  be  useful  for  program  verification  in  general  (Bierhoff, 
2009).  Beckman  et  al.  (2008)  show  that  permissions  can  also  soundly  verify  typestates  in  concurrent  pro¬ 
grams  that  use  Atomic  blocks  and  in  fact  enforce  the  use  of  mutual  exclusion  primitives  where  thread-shared 
data  is  accessed  (Beckman  et  al.,  2008).  Considering  these  recent  developments,  the  author  hopes  and  be¬ 
lieves  that  we  have  only  just  begun  to  explore  the  possibilities  of  using  permissions  to  improve  our  programs. 


1.4  Thesis  Outline 

The  remainder  of  this  dissertation  is  organized  into  8  chapters.  Chapter  2  states  the  thesis  of  this  dissertation 
and  supporting  hypotheses.  Chapter  3  explains  our  approach  in  detail  based  on  two  familial-  examples, 
iterators  and  streams.  Both  are  inspired  by  APIs  in  the  Java  standard  library.  The  iterator  example  shows 
how  even  complex  protocols  can  be  specified  and  gives  an  intuition  for  how  to  check  compliance  to  protocols 
using  access  permissions.  The  stream  example  illustrates  how  we  can  verify  API  implementation  code , 
including  the  thorny  issue  of  how  to  deal  with  inheritance.  This  chapter  also  provides  a  brief  introduction  to 
the  lineal-  logic  connectives  we  use. 

Chapter  4  presents  a  formal  account  of  an  object-oriented  language  and  type  system  based  on  the  ideas 
of  state  refinement  and  access  permissions.  Protocol  checking  amounts  to  proving  linear  logic  predicates 
using  the  conventional  proof  rules  for  that  logic  augmented  with  a  judgment  for  permission  splitting  and 
merging.  The  soundness  proof  for  a  fragment  of  this  type  system  appeal's  in  Bierhoff  and  Aldrich  (2007a). 
It  guarantees  that  declared  protocols  are  never  violated  at  run-time. 

Chapter  5  presents  an  inference  algorithm  for  tracking  polymorphic  permissions.  While  the  type  system 
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mentioned  above  “guesses”  proofs  in  our  augmented  linear  logic,  the  inference  algorithm  can  automatically 
generate  these  proofs.  This  is  a  crucial  step  for  enabling  automated  protocol  checking. 

Chapter  6  describes  how  this  inference  algorithm  has  been  embedded  into  practical  tooling  for  the  Java 
language  that  is  based  on  a  branch-sensitive  dataflow  analysis.  Java  annotations  can  be  used  to  describe 
protocols  and  make  protocol  checking  a  modular  problem  that  can  be  tackled  by  analyzing  each  method 
in  the  program  separately.  The  chapter  discusses  several  extensions  to  the  theory  of  access  permissions 
developed  in  Chapters  4  and  5  to  make  the  tool  more  useful  in  practice  and  shows  how  practical  language 
features  such  as  constructors  can  be  handled. 

Chapter  7  summarizes  case  studies  in  specifying  major  APIs  from  the  Java  standard  library  and  checking 
open-source  codebases  with  the  tool  described  in  Chapter  6.  Specified  APIs  include  the  Java  Collections 
framework  and  the  JDBC  library  that  was  used  as  an  example  throughout  this  chapter.  We  checked  compli¬ 
ance  to  these  API  protocols  in  a  paid  of  the  Apache  Beehive  library  and  checked  iterator-related  protocols  in 
PMD,  an  open-source  bug-finding  tool. 

Finally,  Chapter  8  puts  this  dissertation  in  the  context  of  related  work  and  Chapter  9  concludes  with  a 
discussion  of  the  lessons  to  be  learned  from  this  research  and  future  work  it  motivates. 

Typefaces.  We  use  the  following  formatting  conventions: 

•  Monospace  is  used  for  code  embedded  into  the  text,  such  as  an  if  statement. 

•  Sans  serif  is  used  for  typestates  such  as  open,  although  we  occasionally  use  italics  to  refer  to  the 
conceptual  notion  of  a  state. 

•  Caps  is  used  for  naming  inference  rules. 
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Chapter  2 


Thesis  and  Hypotheses 

2.1  Thesis 

Typestate-based  protocols  with  state  refinement  and  access  permissions  can  be  used  for  auto¬ 
mated,  static,  modular  enforcement  of  API  protocols  in  practical  object-oriented  software. 

This  thesis  sets  the  bar  high  in  claiming  the  applicability  of  our  approach  to  “practical  software”.  What 
we  have  in  mind  with  this  charged  word  is  to  show  that  our  approach  can  be  used  with  common  software 
practices — which  in  particular  include  the  APIs  and  programming  languages  in  practical  use,  as  well  as  how 
code  is  commonly  written — as  opposed  to  only  being  applicable  to  toy  programs.  For  this  dissertation  we 
approximate  this  standard  as  follows:  we  chose  commonly  used  APIs  from  the  Java  standard  library  and 
released  code  from  open-source  software  projects  as  study  subjects  in  several  case  studies. 

We  do  not  believe  that  our  approach  would  work  well  with  every  conceivable  API  or  piece  of  code,  and 
in  particular-  not  with  all  off-the-shelf — found  in  the  wild — code  as-is.  But  we  will  provide  evidence  that 
our  approach  is  well-suited  for  expressing  and  enforcing  challenging  protocols  for  real  APIs,  and  that  it  can 
achieve  good  precision  when  checking  protocols  in  off-the-shelf  code.  Furthermore,  small  code  changes  can 
sometimes  overcome  reasoning  imprecisions. 

Our  approach  therefore  has  properties  similar  to  a  conventional  type  system  such  as  the  one  found  in 
Java:  small  code  changes  sometimes  allow  successful  protocol  checking,  and  runtime  checks  in  the  program 
can  be  used  to  overcome  reasoning  imprecisions,  similar  to  typecasts  in  object-oriented  type  systems. 


2.2  Hypotheses 

In  an  ideal  scenario,  we  would  evaluate  our  approach  by  deploying  it  to  software  engineers  in  real-world 
software  development  projects,  or  by  using  it  on  a  large  number  of  software  artifacts.  But  an  evaluation  on 
this  scale  is  beyond  the  scope  of  this  dissertation. 

Instead  we  chose  to  focus  on  four  testable  hypotheses  that  address  the  predominant  concerns  of  this  re¬ 
search:  they  aim  at  establishing  our  approach’s  well-foundedness  and  its  applicability  to  “practical  software” — 
as  explained  above — in  particular  with  respect  to  APIs,  conventional  programming  languages,  and  how  code 
is  written  in  practice. 
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This  section  discusses  each  of  these  hypotheses  in  detail  and  previews  the  validation  this  dissertation 
provides  for  them.  Our  first  hypothesis  is  that  common  protocols  can  be  captured  succinctly  with  our 
approach,  establishing  applicability  to  APIs.  Our  second  hypothesis  is  that  protocol  compliance  can  be 
checked  in  a  sound  modular  fashion,  showing  our  approach’s  well-foundedness.1  The  third  hypothesis  is 
that  the  approach  can  be  embodied  in  tools  for  conventional  programming  languages  that  impose  annotation 
burden  for  developers  that  is  comparable  to  type  declarations,  establishing  the  approach’s  applicability  to 
commonly  used  languages.  Finally,  the  fourth  hypothesis  is  that  our  approach  can  be  used  to  check  protocols 
in  off-the-shelf  software,  which  shows  the  approach's  applicability  to  code  written  in  practice. 

2.2.1  Capture  Common  Protocols  Succinctly 

Our  approach  requires  relevant  protocols  to  be  specified  before  they  can  be  checked.  We  were  concerned 
whether  interesting  protocols  could  be  specified  in  a  concise  way.  Our  first  hypothesis  therefore  is: 

Typestate -based  specifications  with  access  permissions  can  succinctly  capture  API  protocols 
commonly  occurring  in  practice. 

We  support  this  hypothesis  with  the  following  evidence: 

1.  Case  studies  on  several  Java  APIs  in  wide  practical  use  (Section  7.1)  showing  that  our  approach  can 
express  commonly-used  protocols. 

2.  Identification  of  a  number  of  challenging  recurring  patterns,  all  of  which  can  be  expressed  with  our 
approach  (Section  7.4.1). 

3.  Measurements  showing  that  the  number  of  annotations  needed  for  capturing  protocols  with  our  ap¬ 
proach  is  small,  especially  compared  to  informal  documentation. 

These  results  establish  the  applicability  of  our  approach  to  API  protocols  occurring  in  practice. 

2.2.2  Sound  Modular  Checking 

In  order  to  “statically  enforce  protocols”  as  stated  in  the  thesis  and  therefore  provide  positive  assurance  at 
compile-time  that  protocols  arc  followed  we  have  to  make  sure  that  our  approach  is  well-founded.  Our 
second  hypothesis  expresses  this  concern: 

Typestate -based  protocols  with  access  permissions  can  be  verified  in  a  sound  modular  fashion. 

This  hypothesis  is  supported  by  a  formalization  of  our  approach  as  a  linear  dependent  type  system 
(Chapter  4)  and  a  proof  of  soundness  (Bierhoff  and  Aldrich,  2007a).  The  proof  of  soundness,  informally 
speaking,  guarantees  that  programs  that  typecheck  in  our  type  system  will  never  violate  declared  protocols 
at  run-time. 

The  formalization  characterizes  the  theoretical  properties  of  our  approach,  which  include  soundness, 
modularity,  the  approach’s  character  as  a  refinement  type  system,  its  aliasing  flexibility,  and  its  support  for 
dynamic  state  tests  and  flexible  overriding  in  subclasses. 

'Applicability  to  APIs  precedes  the  well-foundedness  hypothesis  because  without  being  able  to  capture  common  protocols  there 
would  be  no  point  in  checking  them. 
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2.2.3  Automation 

We  were  concerned  to  which  extent  programs  could  be  checked  automatically  for  protocol  compliance. 
In  particular,  if  the  approach  required  manual  proofs  that  programs  are  consistent  with  protocols  then  its 
usability  by  software  developers  would  be  greatly  reduced.  Our  second  hypothesis  is  therefore: 

Access  permission-based  protocol  checking  can  be  automated  with  annotation  burden  compa¬ 
rable  to  conventional  type  declarations  and  used  with  conventional  programming  languages. 

We  support  this  hypothesis  as  follows: 

1 .  We  describe  a  type  inference  system  inspired  by  constraint  logic  programming  that  reduces  tracking 
permissions  through  code  to  linear  constraints  (Chapter  5). 

2.  We  embedded  this  inference  algorithm  into  a  tool.  Plural,  for  checking  protocols  in  the  Java  pro¬ 
gramming  language  (Chapter  6).  The  tool  requires  developers  to  provide  annotations  for  method 
parameters  and  fields,  but  processes  individual  Java  methods  fully  automatically. 

3.  Measurements  showing  that  developer-provided  annotations  arc  in  size  comparable  to  conventional 
Java  typing  information  (both  for  describing  API  protocols  and  for  tracking  objects  from  APIs  in 
client  code). 

4.  Measurements  showing  that  Plural  checks  protocols  on  average  fast  enough  for  interactive  use. 

These  results  emphasize  that  our  approach  is  amenable  to  automated  reasoning,  which  makes  it  interest¬ 
ing  for  use  in  practice. 

2.2.4  Practical  Checking 

As  with  any  sound  approach  to  a  statically  undecidable  problem,  there  will  be  reasoning  imprecisions  which 
can  lead  to  false  positives  when  using  our  approach  for  checking  protocols.  We  were  concerned  that  large 
numbers  of  false  positives  would  make  the  approach  impractical  to  use.  Consequently,  our  final  hypothesis 
is  the  following: 

The  approach  can  enforce  API  protocols  in  off-the-shelf  object-oriented  software. 

We  support  this  hypothesis  with  the  following  evidence: 

1.  Case  studies  showing  that  the  tool  can  check  protocols  of  Java  APIs  in  existing  open-source  codebases 
with  few  false  positives  (Chapter  7). 

2.  A  comparison  between  our  approach  and  the  leading  existing  approach  in  checking  protocol  com¬ 
pliance  under  aliasing  showing  that  our  approach  successfully  checks  protocol  compliance  in  cases 
where  existing  approaches  cannot  (Section  7.3.3). 

3.  A  categorization  of  remaining  false  positives  that  shows  that  most  false  positives  can  be  removed 
with  simple  improvements  to  the  approach  and  tooling.  Furthermore,  refactorings  sometimes  enable 
automated  protocol  checking  in  code  that  could  not  be  analyzed  as  written. 
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These  results  establish  applicability  of  our  approach  to  software  found  in  practice.  They  do  not  mean 
that  it  will  be  straightforward  to  apply  our  approach  to  every  conceivable  piece  of  existing  code,  but  they 
show  that  our  approach  is  flexible  enough  to  check  protocols  in  code  written  to  serve  a  practical  purpose 
with  reasonable  precision. 


Chapter  3 

Approach:  Access  Permissions 


This  chapter  introduces  the  major  elements  of  the  proposed  approach  with  examples  from  the  Java  standard 
library.  Section  3.1,  on  Java  iterators,  focuses  on  specifying  protocols  and  checking  client  code.  Stream 
implementations  in  the  Java  standard  library  arc  used  to  illustrate  checking  of  API  implementation  code 
against  declared  protocols  and  our  handling  of  subtyping  and  inheritance  (Section  3.2). 

This  chapter  neither  attempts  to  evaluate  the  approach  presented  in  this  dissertation,  nor  does  the  order 
in  which  ideas  are  presented  reflect  the  genesis  of  the  approach.  The  chapter  uses  a  series  of  convenient 
examples  to  informally  introduce  the  major  elements  of  the  approach.  Chapter  7  evaluates  the  approach  in 
several  case  studies. 


3.1  Java  Iterators 

This  section  illustrates  specification  of  protocols  and  verification  of  client  code  based  on  a  case  study  with 
Java  iterators  (Bierhoff,  2006).  Iterators  follow  a  straightforward  protocol,  but  define  complicated  alias¬ 
ing  restrictions  that  are  easily  violated  by  developers.  They  are  therefore  a  good  vehicle  to  introduce  our 
approach  to  handling  aliasing  in  protocol  verification. 

3.1.1  Specification  Goals 

The  specification  presented  in  this  section  models  the  Iterator  interface  defined  in  the  Java  standard 
library.  We  will  focus  on  read-only  iterators,  i.e.,  iterators  that  cannot  modify  the  collection  they  iterate 
over.  We  will  refer  to  read-only  iterators  simply  as  “iterators”  and  qualify  full  Java  iterators  as  “modifying 
iterators”.  Full  iterators  will  be  discussed  in  Section  3.1.6.  Goals  of  the  presented  specification  include  the 
following. 

•  Capture  the  usage  protocol  of  Java  iterators. 

•  Allow  creating  an  arbitrary  number  of  iterators  over  collections. 

•  Invalidate  iterators  before  the  iterated  collection  is  modified. 
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Figure  3.1:  Read-only  iterator  state  machine  protocol.  Return  values  for  hasNext  are  included  to  show  the 
possible  outcomes  of  dynamic  state  tests. 

3.1.2  State  Machine  Protocol 

An  iterator  returns  all  elements  of  an  underlying  collection  one  by  one.  Collections  in  the  Java  standard 
library  are  lists  or  sets  of  objects.  Their  interface  includes  methods  to  add  objects,  remove  objects,  and  test 
whether  an  object  is  part  of  the  collection  (see  Figure  3.3).  The  interface  also  defines  a  method  iterator 
that  creates  a  new  iterator  over  the  collection. 

The  Iterator  interface  defines  two  methods  hasNext  and  next.  Every  call  to  next  returns  another 
object  contained  in  the  iterated  collection.  The  method  hasNext  determines  whether  next  can  be  called.  It 
is  illegal  to  call  next  once  hasNext  returns  false.  Figure  3.1  illustrates  this  protocol  as  a  state  machine. 

The  method  hasNext  determines  whether  another  object  is  available  or  the  iteration  reached  its  end.  We 
call  hasNext  a  dynamic  state  test :  its  return  value  indicates  what  state  the  iterator  is  currently  in.  Notice 
that  hasNext  can  always  be  called  and  does  not  change  state.  The  next  section  will  show  how  this  protocol 
can  be  specified. 


3.1.3  Iterator  Interface  Specification 

States  through  refinement.  Following  previous  work  we  call  the  set  of  possible  states  of  an  object  its 
state  space  and  define  it  as  part  of  the  object’s  interface.  As  suggested  above,  we  can  model  the  iterator  state 
space  with  two  states,  available  and  end.  In  our  approach,  states  are  introduced  by  refinement  of  an  existing 
state.  State  refinement  corresponds  to  OR-states  in  Statecharts  (Harel,  1987)  and  puts  states  into  a  hierarchy. 

State  refinement  allows  interfaces  to,  at  the  same  time,  inherit  their  supertypes’  state  spaces,  define  addi¬ 
tional  (more  fine-grained)  states,  and  be  properly  substitutable  as  subtypes  of  extended  interfaces  (Bierhoff 
and  Aldrich,  2005).  Refinement  guarantees  substitutability  because  all  new  states  defined  in  a  subtype  cor¬ 
respond  to  a  state  inherited  from  the  supertype.  States  form  a  hierarchy  rooted  in  a  state  alive  defined  in  the 
root  type  Object.  For  instance,  the  state  space  for  iterators  can  be  defined  as  follows. 

states  available,  end  refine  alive; 

Typestates  do  not  correspond  to  fields  in  a  class.  They  describe  an  object’s  state  of  execution  abstractly 
and  information  about  fields  can  be  tied  to  typestates  using  state  invariants  (see  Section  3.2.1). 
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Access  permissions  capture  design  intent.  Iterators  have  only  two  methods,  but  these  have  very  different 
behavior.  While  next  can  change  the  iterator’s  state,  hasNext  only  tests  the  iterator’s  state.  And  even  when 
a  call  to  next  does  not  change  the  iterator’s  typestate,  it  still  advances  the  iterator  to  the  next  object  in  the 
sequence.  hasNext,  on  the  other  hand,  treats  the  iterator  as  pure :  it  does  not  modify  the  iterator  at  all. 

We  use  a  novel  abstraction,  access  permissions  (“permissions”  for  short),  to  capture  this  design  intent 
as  part  of  the  iterator’s  protocol.  Permissions  arc  associated  with  object  references  and  govern  how  objects 
can  be  accessed  through  a  given  reference  (Boyland,  2003).  For  next  and  hasNext  we  need  only  two  kinds 
of  permissions;  additional  permissions  will  be  introduced  later. 

•  full  permissions  grant  read/write  access  to  the  referenced  object  and  guarantee  that  no  other  reference 
has  write  access  to  the  same  object. 

•  pure  permissions  grant  read-only  access  to  the  referenced  object  but  assume  that  other  permissions 
could  modify  the  object. 

A  distinguished  full  permission  can  co-exist  with  an  arbitrary  number  of  pure  permissions  to  the  same 
object.  This  property  will  be  enforced  when  verifying  protocol  compliance.  In  a  specification  we  write 
perm(x)  for  a  permission  to  an  object  referenced  by  x,  where  perm  is  one  of  the  permission  kinds.  Ac¬ 
cess  permissions  carry  state  information  about  the  referenced  object.  For  example,  “full(7/;/.sj  in  available” 
represents  a  full  permission  for  the  receiver  object,  which  we  refer  to  with  this,  that  is  in  the  available  state. 

Linear  logic  specifications  relate  objects.  Methods  can  be  specified  with  a  state  transition  that  describes 
how  method  parameters  change  state  during  method  execution.  Access  permissions  represent  resources 
that  have  to  be  consumed  upon  usage — otherwise  permissions  could  be  freely  duplicated,  possibly  violating 
other  permissions’  assumptions.  In  particular,  permissions  used  in  a  method  call  have  to  be  taken  away  from 
the  caller  and  replaced  by  whatever  permissions  the  method  returns. 

Therefore,  we  base  protocol  specifications  on  linear  logic  (Girard,  1987).  Linear  logic  treats  facts  as 
resources  that  are  consumed  when  used  to  prove  other  facts.  This  matches  our  intuition  of  permissions  as 
resources.  Methods  arc  specified  with  a  linear  implication  (— o)  from  pre-  to  post-condition.  Most  pre-  and 
post-conditions  shown  in  this  chapter  use  conjunction  ((g))  and  disjunction  (0)  to  relate  permissions.1  In 
certain  cases,  internal  choice  (&,  also  called  additive  conjunction)  has  been  useful  (Bierhoff,  2006).  These 
connectives  represent  the  decidable  multiplicative-additive  fragment  of  linear  logic  (MALL). 

The  next  method  illustrates  that  state  transitions  can  be  non-deterministic:  the  method  makes  an  “in¬ 
ternal  choice”,  meaning  that  a  client  cannot  predict  whether  the  iterator  will  be  in  the  available  or  the  end 
state  when  next  returns.  We  can  express  this  uncertainty  by  using  alive  in  next’s  post-condition.  In  a  State- 
chart,  this  corresponds  to  transitioning  to  an  OR-state  (Figure  3.1).  Furthermore,  next  modifies  the  iterator 
(by  advancing  it  to  the  next  element  in  the  collection.  Thus,  we  can  specify  next  so  that  it  requires  a  full 
permission  in  state  available  and  returns  the  full  permission  in  the  alive  state,  which  we  write  as  follows: 

full  (this)  in  available  — o  full  (this)  in  alive 

Dynamic  state  tests  (like  hasNext)  require  relating  the  (Boolean)  method  result  to  the  state  of  the  tested 
object  (usually  the  receiver).  A  disjunction  of  conjunctions  expresses  the  two  possible  outcomes  of  hasNext 

’“Tensor”  (®)  corresponds  to  conjunction,  “alternative”  (0)  to  disjunction,  and  “lolli”  (— o)  to  implication  in  conventional 
Boolean  logic.  The  key  difference  is  that  linear  logic  treats  known  facts  as  resources  that  are  consumed  when  proving  another  fact. 
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where  each  conjunction  relates  a  possible  method  result  to  the  corresponding  state  of  the  receiver  object: 

pu re(this)  — o  ( result  =  true  <8>  pur e(this)  in  available) 

©  ( result  =  false  ©  pur e(this)  in  end) 

Note  that  we  refer  to  the  method  return  value  with  result.  We  use  the  convention  that  — °  binds  weaker 
than  <g)  and  ©.  Finally,  we  will  often  omit  the  default  state  information  “  in  alive”,  as  in  the  above  pre¬ 
condition. 

These  specifications  enforce  the  characteristic  alternation  of  calls  to  hasNext  and  next:  hasNext  deter¬ 
mines  the  iterator’s  current  state.  If  it  returns  true  then  it  is  legal  to  call  next.  The  iterator  is  in  an  unknown 
state  after  next  returns,  and  another  hasNext  call  determines  the  iterator’s  new  state.  Clients  must  call 
hasNext  at  least  once  before  each  call  to  next  in  order  to  establish  the  available  state  required  for  calling 
next. 

3.1.4  Creating  and  Disposing  Iterators 

Multiple  (independent)  iterators  arc  permitted  for  a  single  collection  at  the  same  time.  However,  the  collec¬ 
tion  must  not  be  modified  while  iteration  is  in  progress.  Standard  implementations  tty  to  detect  violations  of 
this  rule,  so-called  concurrent  modifications,  on  a  best-effort  basis.  But,  ultimately,  Java  programmers  have 
to  make  sure  on  their  own  that  collections  are  not  modified  while  iterated.  Following  Ramalingam  et  al. 
(2002)  we  note  that  “concurrent”  modifications  often  occur  in  single-threaded  programs,  for  instance  when 
new  elements  are  added  to  a  list  during  an  iteration  of  the  existing  items  in  the  list. 

This  section  shows  how  concurrent  modifications  can  be  prevented  by  establishing  aliasing  constraints 
between  iterators  and  their  collections.  As  we  will  see,  this  problem  is  largely  orthogonal  to  specifying  the 
relatively  simple  protocol  for  individual  iterators  that  was  discussed  in  the  previous  section. 

Immutable  access  prevents  concurrent  modification.  Access  permissions  can  guarantee  the  absence  of 
concurrent  modification  (as  explained  above).  The  key  observation  is  that  when  an  iterator  is  created  it 
stores  a  reference  to  the  iterated  collection  in  one  of  its  fields.  This  reference  should  be  associated  with 
a  permission  that  guarantees  the  collection’s  immutability  while  iteration  is  in  progress.  We  include  two 
previously  proposed  permissions  (Boyland,  2003)  into  our  system  in  order  to  properly  specify  collections. 

•  immutable  permissions  grant  read-only  access  to  the  referenced  object  and  guarantee  that  no  refer¬ 
ence  has  read/write  access  to  the  same  object. 

•  unique  permissions  grant  read/write  access  and  guarantee  that  no  other  reference  has  any  access  to 
the  object. 

Thus  immutable  permissions  cannot  co-exist  with  full  permissions  to  the  same  object.  We  can  specify 
the  collection’s  iterator  method  using  these  permissions  as  follows.  Notice  how  it  consumes  or  captures 
the  incoming  receiver  permission  and  returns  an  initial  unique  permission  to  a  fresh  iterator  object. 

public  class  Collection  { 

Iterator  iterator!)  :  immutable(f/tts)  — o  unique(resw/f) 

> 
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Collection  c  =  new 
Iterator  it  =  c .iterator ()■,//  legal 
while(it.hasNext()  &&  ...)  {  //legal 
Object  o  =  it.next();  //legal 
Iterator  it2  =  c.iterator();  //  legal 
while(it2.hasNext())  {  //  legal 
Object  o2  =  it2.next();  //  legal 
/*...*/ } 

} 

if(it.hasNext()  &&  c.size()  ==  3)  {  //  legal 
c.remove(it.next());  //  legal 

if(it.hasNext())  /*...*/  }  // ILLEGAL:  Iterator  access  after  iterated  collection  was  modified 
Iterator  it3  =  c.iterator();  //legal 

Figure  3.2:  Simple  Iterator  client  with  concurrent  modification  error 

It  turns  out  that  this  specification  precisely  captures  Sun’s  Java  standard  library  implementation  of  it¬ 
erators:  iterators  are  realized  as  inner  classes  that  implicitly  reference  the  collection  they  iterate.  Using  an 
immutable  permission  for  this  reference  prevents  concurrent  modifications  (see  below).  If  other  references 
might  modify  the  collection  then  this  specification  does  not  allow  iterators  to  be  created. 

Permission  splitting.  How  can  we  track  permissions?  Consider  a  client  such  as  the  one  in  Figure  3.2.  It 
gets  a  unique  permission  when  first  creating  a  collection.  Then  it  creates  an  iterator  that  captures  an  im¬ 
mutable  permission  to  the  collection.  However,  the  client  later  needs  more  immutable  permissions  to  create 
additional  iterators.  Thus  while  a  unique  permission  is  intuitively  stronger  than  an  immutable  permission 
we  cannot  just  coerce  the  client’s  unique  permission  to  an  immutable  permission  and  pass  it  to  iterator: 
it  would  get  captured  by  the  newly  created  iterator,  leaving  the  client  with  no  permission  to  the  collection  at 
all. 

In  order  to  avoid  this  problem  we  use  permission  splitting  in  our  verification  approach.  Before  method 
calls  we  split  the  original  permission  into  two,  one  of  which  is  retained  by  the  caller.  Permissions  are  split 
so  that  their  assumptions  arc  not  violated.  In  particular,  we  never  duplicate  a  full  or  unique  permission  and 
make  sure  that  no  full  permission  co-exists  with  an  immutable  permission  to  the  same  object.  Some  of  the 
legal  splits  arc  the  following. 

unique(.x)  ^  full(.x)  <8>  pure(x) 
full(.x)  ^  full(.x)  <8>  pure(x) 
fu II (re)  ^  immutable(.x)  <g>  immutable(x) 
immutable(.x)  ^  immutable(.x)  <g>  immutable(x) 

They  allow  the  example  client  in  Figure  3.2  to  retain  an  immutable  permission  when  creating  iterators, 
permitting  multiple  iterators  and  reading  the  collection  directly  at  the  same  time. 

Permission  merging  recovers  modifying  access.  When  splitting  a  full  permission  to  a  collection  into 
immutable  permissions  we  lose  the  ability  to  modify  the  collection.  Intuitively,  we  would  like  to  reverse 
permission  splits  to  regain  the  ability  to  modify  the  collection. 
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interface  Iterator<c : Collection,  k  :  Fract>  { 
states  available,  end  refine  alive 

boolean  hasNextO  : 

pure(this)  ( result  =  true  <g>  pure(this)  in  available) 
©  ( result  =  false  ®  pur e(this)  in  end) 
Object  next()  : 

f  ul  I  (this')  in  available  — o  full(f/zzs) 
void  finalize ()  : 

unique(f/zzs)  — o  immutable(c,  fc) 

> 

interface  Collection  { 

void  add(Object  o)  :  full(f/zzs)  — o  full(f/zzs) 
int  size()  :  pure(this)  —o  result  >  0  (g>  pur e(this) 

//  remove (),  contains ()  etc.  similar 

Iterator< this,  k>  iterator ()  : 

immutable(f/zzs,  fc)  unique(rasnZt) 

> 


Figure  3.3:  Read-only  Iterator  and  partial  Collection  interface  specification 


Such  permission  merging  can  be  allowed  if  we  introduce  the  notion  of  fractions  (Boyland,  2003).  Essen¬ 
tially,  fractions  keep  track  of  how  often  a  permission  was  split.  This  later  allows  the  merging  of  permissions 
(with  known  fractions)  by  adding  together  their  fractions.  A  unique  permission  by  definition  holds  a  full 
fraction  that  is  represented  by  one  (1);  the  exact  values  of  other  fractions  do  not  carry  any  semantic  signif¬ 
icance.  We  will  capture  fractions  as  paid  of  our  permissions  and  write  perm(x ,  fc)  for  a  given  permission 
with  fraction  k.  We  usually  do  not  care  about  the  exact  fraction  and  therefore  implicitly  quantify  over  all 
fractions.  If  a  fraction  does  not  change  we  often  will  omit  it.  Fractions  allow  us  to  define  splitting  and 
merging  rules  such  as  the  following: 


unique(a;,  1) 
full (cc,  fc) 
f  ul  I  (ar,  1/2) 

immutable(;r,  k) 


full (cc,  1/2)  ®  pure(x,  1/2) 
full  (a;,  fc/2)  ®  pure(x,  k/ 2) 
immutable(a:,  1/4)  ®  immutable(a:,  1/4) 
immutable(a:,  fc/2)  ®  immutable(:r,  fc/2) 


For  example,  we  can  split  full(zY,  1/2)  into  full(/f,  1/4)  <g>  pur e(it,  1/4)  and  recombine  them.  Such  rea¬ 
soning  lets  our  iterator  client  recover  a  unique  iterator  permission  after  each  call  into  the  iterator. 


Recovering  collection  permissions.  Iterators  are  created  by  trading  a  collection  permission  for  a  unique 
iterator  permission.  We  essentially  allow  the  opposite  trade  as  well  in  order  to  modify  a  previously  iterated 
collection  again:  We  can  safely  consume  a  unique  iterator  permission  and  recover  the  permissions  to  its 
fields  because  no  reference  will  be  able  to  access  the  iterator  anymore.  We  could  do  this  at  any  time,  but 
a  good  time  to  do  so  is  when  an  iterator  is  no  longer  used.  A  simple  live  variable  analysis  can  identify 
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Collection  c  =  new  . . . 

Iterator  it<c,  l/2>  =  c . iterator () ; 
while (it .hasNext ()  &&  ...)  { 

Object  o  =  it. next (); 

Iterator  it2<c,  l/4>  =  c. iterator () 
while (it2. hasNext ())  { 

Object  o2  =  it2.next(); 

...  > 

> 

if  (it  .hasNext  ()  &&  c.sizeO  ==  3)  { 
c .  remove  (it .  next  ());// kill  iterator 
if (it .hasNext () )  ...  } 

//  it  definitely  dead 
Iterator  it3<c,  l/2>  =  c. iteratorQ ; 


unique(c) 

immutable(c,  1/2)  g  unique(/f) 
immutable(c,  1/2)  g  unique(fi)  in  available 
immutable(c,  1/2)  g  unique(/f) 
immutable(c,  1/4)  (g  unique(d)  g  unique(/f2) 
immutable(c,  1/4)  g)  unique(fi)  g  unique(//2)  in  available 
immutable(c,  1/4)  g  uniquely/)  g  unique(;f2) 

//  it2  dies 

immutable(c,  1/2)  g  unique(/f) 
immutable(c,  1/2)  g  unique(/f)  in  available 
after  next( )  unique(c)  and  no  permission  for  “it” 

II  ILLEGAL 
unique(c) 

immutable(c,  1/2)  g  unique(/fi) 


Figure  3.4:  Verifying  a  simple  Iterator  client 


when  variables  with  unique  permissions  are  no  longer  used.  (As  a  side  effect,  a  permission-based  approach 
therefore  allows  identifying  dead  objects.) 

For  lack  of  a  more  suitable  location,  we  annotate  the  finalize  method  to  indicate  what  happens  when 
an  iterator  is  no  longer  usable.  And  in  order  to  re-establish  exactly  the  permission  that  was  originally 
passed  to  the  iterator  we  parameterize  Iterator  objects  with  the  collection  instance  they  iterate  (c)  and  the 
collection  permission’s  fraction  ( k ).  The  finalize  specification  can  then  release  the  captured  collection 
permission  from  dead  iterators.  Figure  3.3  summarizes  the  complete  specification  for  iterators  and  a  partial 
collection  specification.  The  specifications  augment  the  conventional  Java  interfaces  without  changing  their 
type  signatures. 

3.1.5  Client  Verification 

Figure  3.4  illustrates  how  our  client  from  Figure  3.2  can  be  verified  by  tracking  permissions  and  split¬ 
ting/merging  them  as  necessary.  After  each  line  of  code  we  show  the  current  set  of  permissions  on  the 
right-hand  side  of  the  figure.  We  recover  collection  permissions  by  “killing”  dead  iterators  as  soon  as  possi¬ 
ble  (applying  their  specification  for  finalize).  We  correctly  identify  the  seeded  protocol  violation. 

3.1.6  Modifying  Iterators 

Actual  Java  iterators  can  modify  the  iterated  collection:  the  additional  method  remove  removes  the  most 
recently  retrieved  element  from  the  underlying  collection.  This  invalidates  all  other  iterators  over  the  same 
collection,  but  not  the  iterator  at  hand. 


State  dimensions  for  orthogonal  concerns.  We  propose  state  dimensions  to  address  separate  concerns 
with  states  that  are  independent  from  each  other  (Bierhoff  and  Aldrich,  2005).  State  dimensions  arc  orthog¬ 
onal  sets  of  mutually  exclusive  states  and  correspond  to  AND-states  in  Statecharts  (Hard,  1987). 
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Legend 


AND  state 


dimension  j  dimension  \ 


Atomic 

i 

Atomic 

state 

i 

state 

[condition] 
operati<5n(. . .)  / 


return  value 


Figure  3.5:  Modifying  iterator  state  machine  protocol.  The  shaded  area  is  affected  by  remove. 


We  model  full  Java  iterators  with  three  orthogonal  dimensions  (Figure  3.5).  At  runtime,  an  iterator 
object  will  be  in  exactly  one  of  the  states  from  each  dimension.  Notice  that  we  introduce  dimensions  by 
simply  refining  the  same  state  more  than  once  (Figure  3.6).  The  three  iterator  dimensions  have  the  following 
meaning  (Figure  3.5): 

•  next  captures  whether  another  element  is  available  or  not.  This  corresponds  to  the  state  space  of 
read-only  iterators. 

•  previous  captures  whether  an  element  can  be  removed  or  not.  Notice  that  removing  an  element  is  only 
possible  if  an  element  was  retrieved  but  not  removed  yet. 

•  mode  indicates  whether  the  current  iterator  is  read-only  or  modifying.  Once  created,  iterators  do  not 
change  state  in  this  dimension. 

These  dimensions  encode  8  unique  state  combinations  (there  are  2  states  each  in  3  dimensions).  In  the 
following  we  show  how  dimensions  can  be  used  for  concise  protocol  specifications  and  aliasing  control. 

Permissions  for  dimensions.  Iterator  methods  only  access  certain  state  dimensions:  hasNext  only  looks 
at  the  next  dimension,  remove  only  modifies  the  previous  dimension,  and  next  modifies  next  and  pre¬ 
vious.  We  include  this  information  as  an  additional  parameter  into  access  permissions  and,  e.g.,  write 
fu II (this,  previous)  for  a  permission  for  the  previous  dimension.  It  corresponds  to  an  “area”  in  the  itera¬ 
tor  Statechart  within  which  the  permission  can  change  state.  For  example,  the  shaded  area  in  Figure  3.5 
corresponds  to  the  permission  needed  to  call  remove.  Notice  that  separate  permissions  can  exist  for  orthog¬ 
onal  dimensions. 

Figure  3.6  shows  a  specification  for  full  Java  iterators.  Notice  how  the  different  methods  use  permissions 
for  different  state  dimensions.  In  Section  7.1.2  we  will  see  how  to  specify  the  Collection’s  iterator 
method  so  that  it  leaves  the  decision  whether  an  iterator  is  modifying  or  read-only  up  to  the  client. 

3.2  Java  Stream  Implementations 

I/O  protocols  are  common  examples  for  typestate-based  protocol  enforcement  approaches  (DeLine  and  Fah- 
ndrich,  2001,  2004b;  Bierhoff  and  Aldrich,  2005).  This  section  summarizes  a  case  study  in  applying  our 


3.2.  JAVA  STREAM  IMPLEMENTATIONS 


27 


interface  Iterator<c  :  Iterable,fc  :  Fract>  { 
next  =  available,  end  refine  alive 
previous  =  retrieved,  removed  refine  alive 
mode  =  readonly,  modifying  refine  alive 

boolean  hasNextO  :  pure(f/zzs,  next)  — o 

( result  =  true  (g>  pur e(this,  next)  in  available) 

®  ( result  =  false  <g>  pur e(this,  next)  in  end) 

Object  next()  : 

full(f/zzs,  next)  in  available  ®  full(f/zzs,  previous)  — o 
f  ul  I  (this ,  next)  ®  full(f/zzs,  previous)  in  retrieved 
void  remove ()  : 

full(f/zzs,  previous)  in  retrieved  ®  immutable(t/zzs,  modifying)  — 
f  ul  {[this,  previous)  in  removed  g  immutable(f/zzs,  modifying) 
void  finalizeO  : 

(unique(f/zz'i)  in  readonly  — o  immutable(c,  k)) 

&  (unique(f/zzs)  in  modifying  — o  full(c,  k)) 


Figure  3.6:  Java  modifying  Iterator  specification 


approach  to  Java  character  streams  and  in  particular  stream  pipes  and  buffered  input  streams.  The  section  fo¬ 
cuses  on  implementation  verification  of  stream  classes,  which — to  our  knowledge — has  not  been  attempted 
with  typestates  before.  Implementation  verification  generalizes  techniques  shown  in  the  previous  section  for 
client  verification.  This  dissertation  assumes  that  concurrent  object  accesses  arc  correctly  synchronized.  A 
formal  treatment  of  access  permissions  in  multi-threaded  programs  can  be  found  in  Beckman  et  al.  (2008). 

3.2.1  Stream  Pipes 

Pipes  carry  alphanumeric  characters  from  a  source  to  a  sink.  Pipes  arc  commonly  used  in  operating  system 
shells  to  forward  output  from  one  process  to  another  process.  The  Java  I/O  library  includes  a  pair  of  classes, 
PipedOutputStream  and  PipedlnputStream,  that  offers  this  functionality  inside  Java  applications.  This 
section  provides  a  specification  for  Java  pipes  and  shows  how  the  classes  implementing  pipes  in  the  Java 
standard  library  can  be  checked  using  our  approach. 

Informal  pipe  contract.  In  a  nutshell,  Java  pipes  work  as  follows:  a  character-producing  “writer”  writes 
characters  into  a  PipedOutputStream  (the  “source”)  that  forwards  them  to  a  connected  PipedlnputStream 
(the  “sink”)  from  which  a  “reader”  can  read  them.  The  source  forwards  characters  to  the  sink  using  the 
internal  method  receive.  The  writer  calls  close  on  the  source  when  it  is  done,  causing  the  source  to  call 
receivedLast  on  the  sink  (Figure  3.8). 

The  sink  caches  received  characters  in  a  circular  buffer.  Calling  read  on  the  sink  removes  a  character 
from  the  buffer  (Figure  3.9).  Eventually  the  sink  will  indicate,  using  an  end  of  file  token  (EOF,  -1  in  Java), 
that  no  more  characters  can  be  read.  At  this  point  the  reader  can  safely  close  the  sink.  Closing  the  sink 
before  EOF  was  read  is  unsafe  because  the  source  with  throw  an  exception  when  the  writer  writes  more 
characters  to  the  pipe. 
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Legend 
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Figure  3.7:  PipedlnputStream's  state  space  (inside  open) 


The  pipe  classes  in  Sun’s  standard  library  implementation  have  built-in  runtime  checks  that  throw  ex¬ 
ceptions  in  the  following  error  cases:  (1)  closing  the  sink  before  the  source,  (2)  writing  to  a  closed  source 
or  pushing  characters  to  the  sink  after  the  source  was  closed,  and  (3)  reading  from  a  closed  sink.  The 
specification  we  present  here  makes  these  error  cases  impossible. 

State  space  with  dimensions.  The  source  protocol  can  be  modeled  with  three  states  raw,  open,  and  closed, 
raw  indicates  that  the  source  is  not  connected  to  a  sink  yet.  For  technical  reasons  that  are  discussed  below, 
we  refine  open  into  ready  and  sending.  The  writer  will  always  find  the  source  in  state  ready. 

For  the  sink  protocol  we  again  distinguish  open  and  closed.  A  refinement  of  open  helps  capture 
read’s  protocol:  The  sink  is  in  the  within  state  as  long  as  read  returns  characters;  the  eof  state  has  been 
reached  once  read  returns  the  EOF  token.  While  within,  we  keep  track  of  the  sink’s  buffer  being  empty  or 
nonEmpty.  We  further  refine  nonEmpty  into  partial  and  filled,  the  latter  corresponding  to  a  full  buffer. 

At  the  same  time,  however,  we  would  like  to  keep  track  of  whether  the  source  was  closed,  i.e.,  whether 
receivedLast  was  called.  To  capture  this,  we  refine  nonEmpty  twice,  along  different  dimensions.  We  call 
the  states  for  the  second  dimension  sourceOpen  and  sourceClosed  with  the  obvious  semantics.  Note  that  we 
only  need  the  additional  source  dimension  while  the  buffer  is  nonEmpty;  the  source  is  by  definition  open 
(closed)  in  the  empty  (eof)  state.2  To  better  visualize  the  sink’s  state  space,  Figure  3.7  summarizes  it  as  a 
Statechart. 

Shared  modifying  access.  Protocols  for  source  and  sink  are  formalized  in  Figures  3.8  and  3.9  with  spec¬ 
ifications  that  work  similar  to  the  iterator  example  in  the  last  section.  However,  the  sink  is  conceptually 
modified  through  two  distinct  references,  one  held  by  the  source  and  one  held  by  the  reader.  In  order  to 
capture  this,  we  introduce  our  last  permission. 

•  share  permissions  grant  read/write  access  to  the  referenced  object  and  allow  other  permissions  to  have 
read/write  access  as  well. 

Conventional  programming  languages  can  be  thought  of  as  always  using  share  permissions.  Interest¬ 
ingly,  share  permissions  are  split  and  merged  exactly  like  immutable  permissions.  Since  share  and  im- 

2This  is  only  one  way  of  specifying  the  sink.  It  has  the  advantage  that  readers  need  not  concern  themselves  with  the  internal 
communication  between  source  and  sink. 
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mutable  permissions  cannot  co-exist,  our  rules  force  a  commitment  to  either  one  when  initially  splitting  a 
full  permission.  Examples  include  the  following: 

full (ar,  1/2)  share(a:,  1/4)  ®  share(:r,  1/4) 

share(a;,  k)  share(a;,  /c/2)  ®  share(x,  k/2) 

share(a :,k)  share(:r,  fc/2)  (g)  pure(:r,  fc/2) 

State  guarantees.  We  notice  that  most  modifying  methods  cannot  change  a  stream's  state  arbitrarily.  For 
example,  read  and  receive  will  never  leave  the  open  state. 

To  formalize  this  observation,  we  can  use  the  idea  of  confining  permissions  to  a  part  of  the  state  space 
that  we  introduced  in  Section  3.1.6:  we  can  specify  a  state  guarantee,  i.e.,  a  state  that  cannot  be  left  with  any 
permission.  As  before,  a  state  guarantee  (also  called  the  permission’s  root  state)  corresponds  to  an  "area” 
in  a  Statechart  that  cannot  be  left.  As  such,  state  guarantees  arc  the  result  of  combining  permissions  with 
hierarchical  state  refinement.  They  constrain  the  behavior  of  a  method  to  a  certain  region  in  the  state  space. 
As  an  example,  we  can  write  the  permission  needed  for  read  as  shar e(this,  open).  We  will  usually  omit  the 
trivial  state  guarantee  alive  so  that  for  instance  share(t/?A,  alive)  is  the  same  as  sh  a  re  (//?/. v). 

State  guarantees  turn  out  to  be  crucial  in  making  share  and  pure  permissions  useful  because  they  guar¬ 
antee  the  referenced  object  to  remain  in  the  guaranteed  state  even  after  possible  modifications  through  other 
permissions. 


Explicit  fractions  for  temporary  heap  sharing.  When  specifying  the  sink  methods  used  by  the  source 
(receive  and  receivedLast)  we  have  to  ensure  that  the  source  can  no  longer  call  the  sink  after  received- 
Last  so  the  sink  can  be  safely  closed.  Moreover,  in  order  to  close  the  sink,  we  need  to  restore  a  permission 
rooted  in  alive.  Thus  the  two  share  permissions  for  the  sink  have  to  be  joined  in  such  a  way  that  there  arc 
definitely  no  other  permissions  relying  on  open  (such  permissions,  e.g.,  could  have  been  split  off  of  one  of 
the  share  permissions). 

We  again  rely  on  fractions  to  accomplish  this  task.  We  use  fractions  to  track,  for  each  state  separately, 
how  many  permissions  rely  on  it.  What  we  get  is  a  fraction  function  that  maps  guaranteed  states  (i.e.,  the 
permission’s  root  and  its  super-states)  to  fractions.  For  example,  if  we  split  an  initial  unique  permission  for 
a  PipedlnputStream  into  two  share  permissions  guaranteeing  open  then  these  permissions  rely  on  open 
and  alive  with  a  1/2  fraction  each.3 

In  order  to  close  the  sink,  we  have  to  make  sure  that  there  arc  exactly  two  share  permissions  relying  on 
open.  Fraction  functions  make  this  requirement  precise.  For  reasons  of  readability,  we  use  the  abbreviation 
half  in  Figures  3.8  and  3.9  that  stands  for  the  following  permission. 

ha  If  (re,  open)  =  share(x,  open,  {alive  i— r  1/2,  open  i— r  1/2}) 

By  adding  fractions  and  moving  the  state  guarantee  up  in  the  state  hierarchy,  the  initial  permission  for 
the  sink,  unique/f/tA,  alive,  {alive  i— >  1}),  can  be  regained  from  two  half(t/?A,  open)  permissions.4  half  is 
the  only  permission  with  an  explicit  fraction  function.  All  other  specifications  implicitly  quantify  over  all 
fraction  functions  and  leave  them  unchanged. 

iterator  permissions  are  rooted  in  alive  and  their  fraction  functions  map  alive  to  the  given  fraction. 

4  Any  other  combination  of  fractions,  such  as  1/4  and  3/4,  will  work  as  well. 
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public  class  PipedOutputStream  { 

states  raw,  open,  closed  refine  alive; 
states  ready,  sending  refine  open; 

raw  :=  sink  =  null 
ready  :=  half  (sink,  open) 
sending  :=  sink  null 
closed  :=  sink  yf  null 

private  PipedlnputStream  sink; 

public  PipedOutputStreamO  :  1  — o  unique(f/zzs)  in  raw 

{  > 


void  connect  (PipedlnputStream  snk)  :  full  (this)  in  raw  ®  half (snk,  open)  — °  full  (this)  in  ready 
{  sink  =  snk;  store  permission  in  field 

>  f\i\\(this)  in  ready 

public  void  write(int  b)  :  fu\\(this,  open)  in  ready®  6  >  0  — °  full(f/zzs,  open)  in  ready 
{  half  (sink,  open)  from  invariant 

sink. receive (b)  ;  returns  half(szzz&,  open) 

}  fu\\(this,  open)  in  ready 


public  void  close()  :  full  (this)  in  ready  — o  full  (this)  in  closed 
{ 

sink.receivedLastO  ; 

> 


half(szzzA:,  open  (from  invariant 
consumes  half(sznk,  open) 
full(f/zzs)  in  closed 


Figure  3.8:  Java  PipedOutputStream  (simplified) 


Existing  modular  typestate  checkers  (DeLine  and  Fahndrich,  2001,  2004b)  cannot  handle  the  pipe  ex¬ 
ample  because  objects  cannot  change  state  once  they  arc  shared.  Explicit  fractions  arc  unsatisfying,  and  we 
never  used  them  in  our  case  studies  (Chapter  7).  But  the  pipe  example  suggests  that  they  can  give  crucial 
expressiveness. 


State  invariants  map  typestates  to  fields.  We  now  have  a  sufficient  specification  for  both  sides  of  the 
pipe.  In  order  to  verify  their  implementations  we  need  to  know  what  typestates  correspond  to  in  implemen¬ 
tations.  Our  implementation  verification  extends  Fugue’s  approach  of  using  state  invariants  to  map  states  to 
predicates  that  describe  the  fields  of  an  object  in  a  given  state  (DeLine  and  Fahndrich,  2004b).  In  addition 
to  invariants  for  atomic  states,  we  allow  state  invariants  for  dimensions  and  refined  states  in  order  to  capture 
invariants  common  to  all  substates  of  a  given  state  or  dimension  (Bierhoff  and  Aldrich,  2005).  In  particular, 
conventional  class  invariants  (Leavens  et  ah,  1999)  simply  appear  as  state  invariants  for  alive. 

Figure  3.8  shows  that  the  source’s  state  invariants  describe  its  three  states  in  the  obvious  way  based  on 
the  field  sink.  Notice  that  the  invariant  for  ready  uses  the  permission  half  (sink,  open)  to  control  access 
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class  PipedlnputStream  -f 

stream  =  open,  closed  refines  alive; 

position  =  within,  eof  refines  open; 

buffer  =  empty,  nonEmpty  refines  within; 

filling  =  partial,  filled  refines  nonEmpty; 

source  =  sourceOpen,  sourceClosed  refines  nonEmpty; 

empty  :=  in  <  0  ®  closedByWriter  =  false 
partial  :=  in  >  0  ®  in  yf  out 
filled  :=  in  =  out 

sourceOpen  :=  closedByWriter  =  false 
sourceClosed  :=  closedByWriter  ®  half  (this,  open) 
eof  :=  in  <  0  (g>  closedByWriter  ®  half  (this,  open) 
open  :=  closedByReader  =  false 
closed  :=  closedByReader  =  true 


private  boolean  closedByWriter  =  false; 
private  volatile  boolean  closedByReader  =  false; 
private  byte  buffer []  =  new  byte [1024]; 
private  int  in  =  -1,  out  =  0; 


public  PipedInputStream(PipedOutputStream  src)  : 
full(src)  in  raw  — o  half  (this,  open)  ®  full(src)  in  open 

{  unique(f/z/s)  in  open  ^  half  (this,  open)  ®  half  (this,  open) 

src  .  connect  (this)  ;  consumes  one  half(f/zzs,  open) 

>  half  (this,  open)  ®  full(src)  in  open 


synchronized  void  receive  (int  b)  :  half  (this,  open)  ®  b  >  0  — °  half(f/zzs,  open)  in  nonEmpty 
{  while  (in  ==  out)  ...  //  wait  a  second  half  (this,  open)  in  filled 

half(this,  open)  in  empty  ©  partial 

if (in  <  0)  {  in  =  0;  out  =  0;  } 
buffer [in++]  =  (byte) (b  &  OxFF) ; 

if  (in  >=  buff  er.  length)  in  =  0;  >  ha\f(this,  open)  in  nonEmpty 


synchronized  void  receivedLast  ()  :  half  (this,  open)  — °  1 

{  closedByWriter  =  true;  }  this  is  now  sourceClosed  or  eof 


public  synchronized  int  read()  :  share(this,  open)  — ° 

(result  >  0  ®  share(this,  open))  ©  (result  =  —  1  ®  share(this,  open)  in  eof) 
{  ...  }  //  analogous  to  receive () 


public  synchronized  void  close  ()  :  half  (this,  open)  in  eof  — o  unique(f/zzs)  in  closed 
{  ha  If  (this,  open)  from  eof  invariant  ^  unique(f/zzs,  open) 

closedByReader  =  true; 
in  =  -1;  > 


Figure  3.9:  Java  PipedlnputStreatn  (simplified) 
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through  the  sink  field  just  as  through  local  variables. 

The  sink’s  state  invariants  arc  much  more  involved  (Figure  3.9)  and  define,  e.g.,  what  the  difference 
between  an  empty  (in  <  0)  and  a  filled  circular-  buffer  (in  =  out )  is.  Interestingly,  these  invariants  are 
all  meticulously  documented  in  the  original  Java  standard  library  implementation  for  PipedlnputStream 
(Bierhoff  and  Aldrich,  2005).  The  half  permission  to  itself  that  the  sink  temporarily  holds  for  the  time 
between  calls  to  receivedLast  and  close  can  be  merged  with  the  half  permission  required  by  close  into 
a  unique  permission,  which  allows  dropping  the  open  state  guarantee  and  verifying  that  close  is  indeed 
allowed  to  close  the  sink. 

Verification  with  invariants.  Implementation  checking  assumes  state  invariants  implied  by  incoming  per¬ 
missions  and  tracks  changes  to  fields.  Objects  have  to  be  in  a  state  whenever  they  yield  control  to  another 
object,  which  means  during  method  calls  and  returns.  This  means  that  the  source  has  to  transition  to  a  state, 
called  sending  in  Figure  3.8,  before  calling  the  sink.  Such  intermediate  or  hidden  states  help  us  deal  with 
reentrant  calls  (details  in  Section  4.2.2).  We  call  sending  an  intermediary  state  because  the  writer  always 
finds  the  source  ready  and  never  in  the  sending  state. 

Figures  3.8  and  3.9  show  how  implementation  checking  proceeds  for  most  of  the  source’s  and  sink’s 
methods.  Note  that  1  denotes  a  trivially  true  predicate.  We  show  in  detail  how  field  assignments  change 
the  sink’s  state.  The  sink’s  state  information  is  frequently  a  disjunction  of  possible  states.  Dynamic  tests 
essentially  rule  out  states  based  on  contradictory  predicates  (that  result  in  false).  All  of  these  tests  are 
present  in  the  original  Java  implementation;  we  removed  additional  non-null  and  state  tests  that  are  obviated 
by  our  approach.  This  not  only  shows  how  our  approach  forces  necessary  state  tests  but  also  suggests  that 
our  specifications  could  be  used  to  generate  such  tests  automatically,  which  we  leave  to  future  work. 

3.2.2  Buffered  Input  Streams 

A  Buf  f  eredlnputStream  (“buffer”)  wraps  another  “underlying”  stream  and  provides  buffering  of  charac¬ 
ters  for  more  efficient  retrieval.  We  will  use  this  example  to  illustrate  our  approach  to  handling  inheritance. 
Compared  to  the  original  implementation,  we  made  fields  “private”  in  order  to  illustrate  calls  to  overridden 
methods  using  super.  We  omit  intermediate  states  in  this  specification  for  brevity. 

Class  hierarchy.  Buff  eredlnputStream  is  a  subclass  of  FilterlnputStream,  which  in  turn  is  a  sub¬ 
class  of  InputStream.  InputStream  is  the  abstract  base  class  of  all  input  streams  and  defines  their  protocol 
with  informal  documentation  that  we  formalize  in  Figure  3.10.  It  implements  convenience  methods  such  as 
read(int  [] )  in  terms  of  other — abstract — methods.  FilterlnputStream  holds  an  underlying  stream  in 
a  field  s  and  simply  forwards  all  calls  to  that  stream  (Figure  3.10).  Buf  f  eredlnputStream  overrides  these 
methods  to  implement  buffering. 

Frames.  The  buffer  occasionally  calls  overridden  methods  to  read  from  the  underlying  stream.  How 
can  we  reason  about  these  internal  calls?  Our  approach  is  based  on  Fugue’s  frames  for  reasoning  about 
inheritance  (DeLine  and  Fahndrich,  2004b).  Objects  are  divided  into  frames,  one  for  each  class  in  the 
object’s  class  hierarchy.  A  frame  holds  the  fields  defined  in  the  corresponding  class.  We  call  the  frame 
corresponding  to  the  object’s  runtime  type  the  virtual  frame,  referred  to  with  normal  references  (such  as  *, 
y,  and  this).  Inside  a  method,  we  refer  to  the  current  frame — corresponding  to  the  class  that  the  method 
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public  abstract  class  InputStream  { 
states  open,  closed  refine  alive; 
states  within,  eof  refine  open; 

public  abstract  int  read()  : 

share^/z/sf,.,  open)  — o  ( result  >  0  ©  share(f/zz.qv,  open)) 

©  ( result  =  —  1  ©  share(f/zzsfn  open)  in  eof) 
public  abstract  void  close ()  : 

full(f/z«fn  alive)  in  open  — o  full(f/zz%,  alive)  in  closed 

public  int  read (byte  []  buf)  : 

share(f/zzs,  open)  ©  buff  null  — o  ( result  >  0  ©  share(f/zzs, open)) 
©  ( result  =  —  1  ©  share(f/zzs,  open)  in  eof) 

{  ...  for ( . . . ) 

int  c  =  this.readO  ...  } 

> 

public  class  FilterlnputStream  extends  InputStream  { 
within  :=  unique(s)  in  within 
eof  :=  unique(s)  in  eof 
closed  :=  unique(s)  inclosed 

private  volatile  InputStream  s; 

protected  FilerlnputStreamflnputStream  s)  : 

unique(s,  alive)  in  open  — o  unique(f/zz5fr,  alive)  in  open 
{  this.s  =  s;  } 

...  //  readQ  and  close ()  forward  to  s 


Figure  3.10:  Java  FilterlnputStream  forwards  all  calls  to  underlying  InputStream  (simplified) 
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Figure  3.11:  Frames  of  a  Buff  eredlnputStream  instance  in  stat  e  filled.  The  shaded  virtual  frame  is  in  a 
different  state  than  its  super- frame. 


is  defined  in — with  thisfr,  and  the  frame  corresponding  to  the  immediate  superclass  is  called  super  frame. 
Figure  3.11  shows  a  sample  Buff  eredlnputStream  instance  with  its  three  frames. 


Frame  permissions.  We  interpret  permissions  to  grant  access  to  a  particular  frame.  The  permissions  we 
have  seen  so  far  give  a  client  access  to  the  referenced  object’s  virtual  frame.  Permissions  for  other  frames 
are  only  accessible  from  inside  a  subclass  through  super. 

Figure  3.11  illustrates  that  a  Buff  eredlnputStream’s  state  can  differ  from  the  state  its  Filterlnput- 
Stream  (“filter”)  frame  is  in:  the  filter’s  state  might  be  eof  (when  the  underlying  stream  reaches  eof)  while 
the  buffer’s  is  still  within  (because  the  buffer  array  still  holds  unread  characters).  The  state  invariants  in 
Figure  3. 12  formalize  this.  They  let  us  verify  that  super  calls  in  the  buffer  implementation  respect  the 
filter’s  protocol.  Notice  that  the  buffer  methods  call  their  overridden  filter  methods  only  sometimes  (Figure 
3.12).  Our  approach  of  decoupling  states  of  different  frames  lets  us  verify  this  “selective”  calling  pattern. 

Because  the  states  of  frames  can  differ  it  is  important  to  enforce  that  a  permission  is  only  ever  used  to 
access  fields  in  the  frame  it  grants  permission  to.  In  specifications  we  specifically  mark  permissions  that  will 
actually  access  fields  (and  not  just  call  other  methods)  of  the  receiver  with  tliisfr.  We  require  all  methods  that 
use  these  permissions  to  be  overridden.5  On  the  other  hand,  convenience  methods  such  as  read(int  [] ) 
can  operate  with  permissions  to  the  virtual  frame  and  need  not  be  overridden  (Figure  3.10). 

This  distinction  implies  that  fill  (Figure  3.12)  cannot  call  read(int  [] )  (because  it  does  not  have  a 
suitable  virtual  frame  permission)  but  only  super  ,read().  This  is  imperative  for  the  correctness  of  fill 
because  a  dynamically  dispatched  call  would  lead  back  into  the — still  empty — buffer,  causing  an  infinite 
loop.  (One  can  trigger  exactly  this  effect  in  the  Java  6  implementation  of  Buff  eredlnputStream.) 


Tn  the  trivial  case,  the  overriding  method  would  simply  call  the  overridden  method  with  a  super- call. 
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public  class  Buf f eredlnputStream  extends  FilterlnputStream  { 
states  depleted,  filled  refine  within; 

closed  :=  unique(sMper)  in  closed  ©  buf  =  null 

open  :=  unique(/?zz/)  ©  buf. length  >  0  ©  count  <  buf. length  <g> pos  >  0 
filled  :=  pos  <  count  ©  unique(szzper)  in  open 
depleted  :=  pos  =  count  ©  unique(swper)  in  within 
eof  :=  pos  =  count  ©  unique(szzper)  in  eof 

private  byte  buf  []  =  new  byte  [8192]; 
private  int  count  =  0,  pos  =  0; 


public  Buf f eredInputStream(InputStream  s)  : 
unique(s)  in  open  — o  unique(f/z/ifr)  in  open 

{ 

super (s) ; 

> 

public  synchronized  int  read()  { 
if (pos  >=  count) 

{ 

f  ill  0 ; 

if (pos  >=  count) 
return  -1; 

> 

return  buf[pos++]  &  OxFF; 

> 

private  void  fill()  : 

share(f/!zsfr ,  open)  in  depleted  ©  eof  — o  share(f/zzsfn 

{ 

count  =  pos  =  0; 
int  b  =  super . read() ; 
while (b  >=  0)  { 

buf [count ++]  =  (byte)  (b  &  OxFF); 
if (count  >=  buf . length)  break; 
b  =  super . read () ; 

> 

> 

public  synchronized  void  close  ()  { 
buf  =  null; 
super. close () ; 


count  =  pos  =  0  ©  unique(Z?z//) 
unique(sMper)  in  open 
unique(f/zwfr,  alive)  in  open 


share(f/zz.qv,  open)  in  depleted  ©  eof 
share(f/zwfn  open)  in  filled  ©  eof 

returns  share(f/zz.?fn  open)  in  eof 
any  path:  share(f/z«fn  open)  in  filled 

share(f/zz%,  open)  in  filled  ©  eof 

)  in  filled  ©  eof 

invariant:  uniqu e(super)  in  within  ©  eof 
note:  assumes  buffer  was  fully  read 
unique(itzper)  in  within  ©  eof 
unique(itzper)  in  within 
share(f/zz'ifr,  open)  in  filled 

uniqu e(super)  in  within  ©  eof 
if  loop  never  taken,  share(f/z«fr,  open)  in  eof 
share(f/zzs,  open)  in  filled  ©  eof 

invariant:  unique(stzper)  in  open 
unique(sMy>er)  in  closed 
full(t/zzsfr,  alive)  in  closed 


Figure  3.12:  Buf  f  eredlnputStream  caches  characters  from  FilterlnputStream  base  class 
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3.3  Summary 

This  chapter  first  presented  a  specification  of  Java  iterators.  Our  approach  is  to  associate  collections  and 
iterators  with  access  permissions,  define  a  simple  state  machine  to  capture  the  iterator  usage  protocol,  and 
track  permission  information  using  a  decidable  fragment  of  linear  logic.  Our  logic-based  specifications 
can  relate  objects  to  precisely  specify  method  behavior  in  terms  of  typestates  and  support  reasoning  about 
dynamic  tests. 

This  chapter  also  showed  that  our  approach  can  verify  that  realistic  Java  pipe  and  buffered  input  stream 
implementations  implement  the  declared  protocol.  Overall,  we  introduced  five  different  permissions  (Table 
1.1).  While  three  arc  adapted  from  existing  work  (Boyland,  2003;  DeLine  and  Fahndrich,  2004b),  we 
proposed  full  and  pure  permissions  (Bierhoff,  2006).  Permission  splitting  and  joining  is  flexible  enough  to 
model  temporary  aliasing  on  the  stack  (during  method  calls)  and  in  the  heap  (e.g.,  in  pipes  and  iterators).  We 
handle  inheritance  based  on  frames  and  permit  dynamic  dispatch  within  objects  for  convenience  methods. 

In  this  chapter  we  chose  to  make  the  linear  logic  formalism  underlying  our  approach  explicit.  Our  tool 
provides  annotations  that  shield  users  from  the  complexities  of  linear  logic  in  most  all  cases  (Chapter  6). 

The  following  chapter  will  treat  the  approach  formally  and  chapters  5  and  6  show  how  reasoning  with 
the  approach  can  be  automated  in  a  tool. 


Chapter  4 

Type  System 


This  chapter  presents  a  formal  treatment  of  state  refinements  and  access  permissions  as  introduced  in  the 
previous  chapter  in  a  linear  dependent  type  system.  The  proof  of  soundness  for  a  fragment  of  the  presented 
calculus  appeal's  as  a  technical  report  (Bierhoff  and  Aldrich,  2007a)  and  is  summarized  in  section  4.2.6.  A 
previous  version  of  this  chapter  appeared  at  the  OOPSLA  2007  conference  (Bierhoff  and  Aldrich,  2007b, 
sections  4  and  5) 

4.1  Formal  Language 

This  section  formalizes  an  object-oriented  language  with  protocol  specifications.  We  briefly  introduce  ex¬ 
pression  and  class  declaration  syntax  before  defining  state  spaces,  access  permissions,  and  permission-based 
specifications.  Finally,  we  discuss  handling  of  inheritance  and  enforcement  of  behavioral  subtyping. 

4.1.1  Syntax 

Figure  4. 1  shows  the  syntax  of  a  simple  class-based  object-oriented  language.  The  language  is  inspired  by 
Featherweight  Java  (FJ,  Igarashi  et  ah,  1999);  we  will  extend  it  to  include  field  assignments  and  typestate 
protocols  in  the  following  subsections.  We  identify  classes  ( C ),  methods  (m),  and  fields  (/)  with  their 
names.  As  usual,  x  ranges  over  variables  including  the  distinguished  variable  this  for  the  receiver  object. 
We  use  an  overbar  notation  to  abbreviate  a  list  of  elements.  For  example,  x  :  T  stands  for  x\  :Tj ,  . . . ,  xn:Tn. 
Types  (T)  in  our  system  include  Booleans  (bool)  and  classes. 

Programs  are  defined  with  a  list  of  class  declarations  and  a  main  expression.  A  class  declaration  CL 
gives  the  class  a  unique  name  C  and  defines  its  fields,  methods,  typestates,  and  state  invariants.  A  constructor 
is  implicitly  defined  with  the  class’s  own  and  inherited  fields.  Fields  (F)  are  declared  with  their  name  and 
type.  Each  field  is  mapped  into  a  part  of  the  state  space  n  that  can  depend  on  the  field  content  (details 
in  Section  4.2.2).  A  method  ( M )  declares  its  result  type,  formal  parameters,  specification,  and  a  body 
expression.  State  refinements  R  will  be  explained  in  the  next  section;  method  specifications  M S  and  state 
invariants  N  arc  deferred  to  Section  4.1.4. 

We  syntactically  distinguish  pure  terms  t  and  possibly  effectful  expressions  e.  Arguments  to  method 
calls  and  object  construction  arc  restricted  to  terms.  This  simplifies  reasoning  about  effects  (Mandelbaum 
et  al.,  2003;  Chin  et  ah,  2005b)  by  making  execution  order  explicit. 
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programs 

PR  ::  = 

<■ CL,e ) 

class  decl. 

CL  ::  = 

class  C  extends  C7  {FRINM} 

field  decl. 

F  ::  = 

/ 

T  in  n 

meth.  decl. 

M  ::  = 

T 

m(T  x)  :  MS  =  e 

state  decl. 

R  ::  = 

d 

=  s  refines  so 

terms 

t  ::  = 

X 

|  0  true  |  false 

| 

t\  and  t‘2  1 1  or  t,2  not  t 

expressions 

e  ::  = 

t 

/  |  assign  /  :=  t 

| 

new  C(t)  |  t.Q.m(t)  super. m(t) 

| 

if(t,  61,62)  let  r  =  ei  in  e2 

values 

v  ::  = 

0 

|  true  false 

references 

r  ::  = 

X 

1  /  I  0 

types 

T  ::  = 

C 

|  bool 

nodes 

n  ::  = 

s 

1  d 

assumptions 

A  ::  = 

n 

|  A\  <g)  A 2  |  Ai  ©  A2 

classes 

C  fields 

f 

variables  x  objects  0 

methods 

m  states 

s 

dimensions  d 

Figure  4.1:  Core  language  syntax.  Specifications  (/,  N,  MS)  arc  in  Figure  4.3. 


Notice  that  we  syntactically  restrict  field  accesses  to  fields  of  the  receiver  class.  Explicit  “getter”  and 
“setter”  methods  can  be  defined  to  give  other  objects  access  to  fields.  Assignments  replace  the  previous 
value  of  a  field  with  the  given  one  and  evaluate  to  the  previous  field  value.1 


4.1.2  State  Spaces 

State  spaces  are  formally  defined  as  a  list  of  state  refinements  (see  Figure  4.1).  A  state  refinement  ( R ) 
refines  an  existing  state  in  a  new  dimension  with  a  set  of  mutually  exclusive  sub-states  (these  concepts  arc 
informally  introduced  in  Chapter  3).  We  use  s  and  d  to  range  over  state  and  dimension  names,  respectively. 
A  node  n  in  a  state  space  can  be  a  state  or  dimension.  State  refinements  arc  inherited  by  subclasses.  We 
assume  a  root  state  alive  that  is  defined  in  the  root  class  Object. 

We  define  a  variety  of  helper  judgments  for  state  spaces  in  Figure  4.2.  refinements(C')  determines  the  list 
of  state  refinements  available  in  class  C.  C  F  A  wf  defines  well-formed  state  assumptions.  Assumptions  A 
combine  states  and  are  defined  in  Figure  4.3.  Conjunctive  assumptions  have  to  cover  orthogonal  parts  of  the 
state  space.  C  F  n  <  n!  defines  the  substate  relation  for  a  class.  C  \~  A#  A'  defines  orthogonality  of  state 
assumptions.  A  and  A'  arc  orthogonal  if  they  refer  to  different  (orthogonal)  state  dimensions.  C  F  A  -<  n 
defines  that  a  state  assumption  A  only  refers  to  states  underneath  a  root  node  n.  C  F  A  <C  n  finds  the 
tightest  such  n. 


’This  semantics  is  appealing  in  the  context  of  linear  type  theory,  but  our  tool  uses  the  conventional  Java  semantics  for  assign¬ 
ments  (cf.  Chapter  6). 
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class  C  extends  C'  {  F  R  . . .  }  refinements^^  =  R! 
refinements(Object)  =  •  refinements(C')  =  R',R 

n  in  refinements(C')  C  F  A\  wf  C  F  A2  wf  C  F  A\  wf  A\  #  A2  C  F  A2  wf 
C  L  n  wf  C  F  A\  ©  A-2  wf  C  F  A\  ®  A2  wf 

d  =s  refines  s  £  refinements^)  C  F  n  wf  C  L  n  <  n"  C  L  n"  <  n 

C  \~  Si  <  d  C  \~  d  <  s  C  h-  n  <  n  C  h  n  <n 

d  =s  refines  s*  £  refinements(C')  d'  =  s'  refines  s*  G  refinements(C)  d  7^  d' 

CL  d#d' 

Chni<!j;  C  b  ni  #  n'2  C  h  n2  <  n'2  C  L  A'  #  A  CF  Ax#  A  CL  A2#  A 
C  L  n\  #  n2  C  L  A  #  A'  CL  A\  <S)  A2  #  A 

C  L  A\  #  A  CL  A2  #  A  C  h  n'  <  n  C  h  ^4i,2  X  n  CL  A\  ®  ^2  wf 

C  L  A\®  A2#  A  C  L  n'  -<  n  C  L  A\®  A2  A  n 

C  h  A\)2  An  (7byli©2l2wf  C  L  A  A  n  Vn/  :  C  L  A  A  n  implies  n  <  n 

C  L  A\®  A2  A  n  Chl<B 

Figure  4.2:  State  space  judgments  (assumptions  A  defined  in  Figure  4.1) 

4.1.3  Access  Permissions 

Access  permissions  p  give  references  permission  to  access  an  object.  Permissions  to  objects  are  written 
access(r,  n,  g,  k,  A)  (Figure  4.3).  We  wrote  perm# ,  n,  g )  in  A,  where  perm  G  {unique,  full,  share,  immutable,  pure}, 
in  the  previous  chapter.  The  additional  parameter  k  allows  us  to  uniformly  represent  all  permissions  as  ex¬ 
plained  below. 

•  Permissions  arc  granted  to  references  r.  References  can  be  variables,  run-time  locations,  and  fields. 

•  Permissions  apply  to  a  particular  subtree  in  the  space  space  of  r  that  is  identified  by  its  root  node 
n.  It  represents  a  state  guarantee  (Section  3.2).  Other  pa  its  of  the  state  space  arc  unaffected  by  the 
permission. 

•  The  fraction  function  g  tracks  for  each  node  on  the  path  from  n  to  alive  a  symbolic  fraction  (Boyland, 

2003).  The  fraction  function  keeps  track  of  how  often  permissions  were  split  at  different  nodes  in  the 
state  space  so  they  can  be  coalesced  later  (see  Section  4.2.5). 

•  The  below  fraction  k  encodes  the  level  of  access  granted  by  the  permission,  k  >  0  grants  modifying 
access,  k  <  1  implies  that  other  potentially  modifying  permissions  exist.  Fraction  variables  z  arc 
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conservatively  treated  as  a  value  between  0  and  1,  i.e.,  0  <  z  <  1. 

•  An  state  assumption  A  expresses  state  knowledge  within  the  permission’s  subtree.  Only  full  permis¬ 
sions  can  permanently  make  state  assumptions  until  they  modify  the  object’s  state  themselves.  For 
weak  permissions,  the  state  assumption  is  temporary ,  i.e.,  lost  after  any  effectful  expression  (because 
the  object’s  state  may  change  without  the  knowledge  of  r). 


We  can  encode  unique,  full,  share,  and  pure  permissions  as  follows. 


unique(r,  n,  g )  in  A 

full (r,n,g)  in  A 
share(r,  n,  g,  k)  in  A 
pure(n,  n,  g)  in  A 


=  access (r,n,  {g,n 
=  access(r,  n,  g,  1,  A) 
=  access (r,n,  g,  k,  A) 
=  access(r,  n,g,0,A) 


1},  1,-4) 

(0  <  k  <  1) 


One  way  of  thinking  about  this  encoding  is  that  we  have  two  kinds  of  fractions  in  a  permission:  (a) 
fractions  that  keeps  track  of  all  splits  and  must  be  strictly  positive,  and  (b)  one  fraction  that  keeps  track  of 
all  splits  except  when  pure  permissions  are  split  off.  Kind  (b)  of  fractions  is  set  to  0  in  pure  permissions. 
This  is  needed  to  distinguish  unique  and  full:  if  kind  (a)  is  1,  we  have  a  unique;  if  (b)  is  1  we  have  a  full. 
Kind  (a)  of  fractions  is  kept  separately  for  different  nodes  in  the  state  hierarchy  in  the  “fraction  function”; 
(b)  is  the  “below  fraction”.  Thus,  the  fraction  function  prevents  against  “loosing  track”  of  pure  permissions. 

In  our  formal  treatment  we  omit  immutable  permissions,  but  it  is  straightforward  to  encode  them  with 
an  additional  “bit”  that  distinguishes  immutable  and  share  permissions.  The  encoding  for  unique  permits 
separate  unique  permissions  for  orthogonal  state  dimensions,  which  is  more  fine-grained  “unique”  access 
than  a  conventional  linear-  reference  to  an  entire  object  (Wadler,  1990). 


4.1.4  Permission-Based  Specifications 

We  combine  atomic  permissions  (p)  and  facts  about  Boolean  values  (q)  using  linear-  logic  connectives  (Fig¬ 
ure  4.3).  We  also  include  existential  ( 3z  :  H.P )  and  universal  quantification  of  fractions  ("iz  :  H.P)  to 
alleviate  programmers  from  writing  concrete  fraction  functions  in  most  cases.  We  type  all  expressions  as  an 
existential  type  ( E ). 


Method  specifications.  Methods  are  specified  with  a  linear-  implication  (— o)  of  predicates  (M S ).  The  left- 
hand  side  of  the  implication  (method  pre-condition)  may  refer  to  method  receiver  and  formal  parameters. 
The  right-hand  side  (post-condition)  existentially  quantities  the  method  result  (a  similar-  technique  is  used  in 
Vault  (DeLine  and  Fahndrich,  2001)).  We  refer  to  the  receiver  with  this  and  call  the  return  value  result. 


State  invariants.  We  decided  to  use  linear-  logic  predicates  for  state  invariants  as  well  ( N ).  In  general, 
several  of  the  defined  state  invariants  will  have  to  be  satisfied  at  the  same  time.  This  is  due  to  our  hierarchical 
state  spaces.  Each  class  declares  an  initialization  predicate  together  with  a  start  state  (/)  that  are  used  for 
object  construction  (instead  of  an  explicit  constructor).  The  initialization  predicate  must  satisfy  the  start 
state’s  invariant. 
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permissions 

P 

:=  access  (r,n,g,k,A) 

facts 

q 

:=  t  =  true  t  =  false 

fraction  fct. 

g 

:=  a  \  n  r->  k 

1  g/ 2  |  51,92 

fractions 

k  : 

:=  1  |  0  |  a  |  jfe/2 

predicates 

P  : 

:=  P  I  q 

I  Pl^P-2  |  1 

I  P1&P2  |  T 

Pi®P2  |  0 

3z  :  H.P  |  Mz  :  H.P 

method  specs 

MS  : 

:=  P^>  E 

expr.  types 

E  : 

:=  3x  :  T.P 

state  inv. 

N  : 

:=  n  =  P 

initial  state 

I  : 

:=  initially  (P,  s  1  <8>  . . 

fract.  terms 

h 

:=  9  I  k 

fract.  types 

H 

:=  Fract  |  n  — ►  Fract 

fract.  vars. 

z 

Figure  4.3:  Permission-based  specifications  (using  syntactic  forms  from  Figure  4.1) 

4.1.5  Handling  Inheritance 

Recall  from  Section  3.2.2  that  permissions  give  access  to  a  particular  frame,  usually  the  virtual  frame  of  an 
object.  Because  of  subtyping,  the  precise  frame  referenced  by  an  object  permission  is  statically  unknown. 
We  will  call  permissions  to  the  virtual  frame  object  permissions.  Because  it  is  unknown  what  frame  an 
object  permission  gives  access  to,  and  different  frames  can  be  in  different  states  (see  Section  3.2.2),  we 
cannot  use  object  permissions  to  access  fields. 

references  r  ::=  . . .  |  super  |  thisfr 

In  order  to  handle  inheritance,  we  distinguish  references  to  the  receiver’s  “current”  frame  ( this fr)  and  its 
super-frame  (super).  Permissions  for  these  “special”  references  arc  called  frame  permissions. 

A  thisf,  permission  grants  access  to  the  fields  in  the  current  frame  and  can  be  used  in  method  specifi¬ 
cations.  thisfr  permissions  cannot  be  used  to  call  methods.  On  the  other  hand,  object  permissions  can  be 
used  for  calling  methods  specified  with  this fr  permissions.  In  order  for  this  coercion  to  work,  all  methods 
requiring  a  this fr  permission  must  be  overridden.  This  guarantees  that  object  permissions  used  in  calling 
such  methods  arc  used  by  the  method  to  access  fields  in  the  virtual  frame,  which  the  object  permission  gives 
access  to.  Permissions  for  super  arc  needed  for  super-calls  and  are  only  available  in  state  invariants.  Note 
that  for  simplicity,  all  fields  arc  treated  as  “private”:  they  can  only  be  accessed  from  inside  the  class  that 
declares  them. 

4.1.6  Behavioral  Subtyping 

Subclasses  should  be  allowed  to  define  their  own  specifications,  e.g.  to  add  precision  or  support  additional 
behavior  (Bierhoff  and  Aldrich,  2005).  However,  subclasses  need  to  be  behavioral  subtypes  (Liskov  and 
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(z:H)€  F 


FPk:  Fract 


T  F  z  :  H  F  F  1  :  Fract  T  F  0  :  Fract  T  F  k/2  :  Fract 

FPk:  Fract  {k  ^  0)  F  F  g  :  n  — >  Fract  rhj:n->  Fract  F  F  (j  :  n'  — >  Fract 


rhnHFn->  Fract  T  F  g/2  :  n  — »  Fract 


r  F  g,(ji  :  n,n'  —*  Fract 


ThriC 

C  \~  A  A  n  r  h  <7  :  n  — *  Fract  n  =  {«//  nodes  between  alive  and  n  inclusive}  F  h  k  :  Fract 

T  h  access(r,  n,  g,  k.  A)  wf 


Figure  4.4:  Fraction  typing  and  well-formed  permissions 


T-Var 

(x  :  T)  G  T 

r  F  x  :  T 


T-LOC 

(o  :  C)  E  T 

rho:C 


T-True  T-False 


r  F  true  :  bool  F  F  false  :  bool 


T-And  T-Or  T-Not 

r  F  t\  :  bool  r  F  f2  :  bool  T  F  t\  :  bool  F  F  f2  :  bool  FF  F  bool 


r  F  and  f2  :  bool 


T  F  ti  or  t2  :  bool 


r  F  not  t  :  bool 


T-Sub 

r  F  f  :  C'  C'  extends  C 


F  F  t  :  C 


Figure  4.5:  Term  typechecking 


Wing,  1994)  of  the  extended  class.  Our  system  enforces  behavioral  subtyping  in  two  steps.  Firstly,  state 
space  inheritance  conveniently  guarantees  that  states  of  subclasses  always  correspond  to  states  defined  in 
superclasses  (Bierhoff  and  Aldrich,  2005).  Secondly,  we  make  sure  that  every  overriding  method’s  specifica¬ 
tion  implies  the  overridden  method’s  specification  (Bierhoff  and  Aldrich,  2005)  using  the  override  judgment 
(Figure  4.7)  that  is  used  in  checking  method  declarations.  This  check  leads  to  method  specifications  that  arc 
contra-variant  in  the  domain  and  co-variant  in  the  range  as  required  by  behavioral  subtyping. 


4.2  Modular  Typestate  Verification 

This  section  describes  a  static  modular  typestate  checking  technique  for  access  permissions  similar  to  con¬ 
ventional  typechecking.  It  guarantees  at  compile-time  that  protocol  specifications  will  never  be  violated  at 
run-time.  We  emphasize  that  our  approach  does  not  require  tracking  typestates  at  run-time. 
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P-Term  P-Field 

r  P  t  :  T  T;  Ah  [t/x\P  localFields(C)  =  f  :  T  T;  Ah  [fi/x\P 

r;  A  h°  i  :  3x  :  T.P  r;  A  h°c  fc  :  3x  :  T~P 

P-New 

rhtTT  init(c)  =  (vJTt.p,a)  r;Ah[t//]p 

T;  A  h°  new  C(t)  :  3x  :  C. access(  x,  alive,  {alive  i — >  1  { ,  1 ,  A) 

P-lF 

(r,  t  =  true);  A  F*  e\  :  3x  :  T.P\  \£i  T  h  t  :  bool  (T,  t  =  false);  A  H  e2  :  3x  :  T.P2  \  £ 2 

r;  A  hivj  if  (t,  ei,  e2)  :  3x  :  T.PX  ©  P2  \  £  1  U  £2 

P-Let 

r;AP  ei  :  3x  :  T.P  \  £x  (r,x  :  T);  (A',P)  P  e2  :  P2  \  £:2 
i  =  1  implies  no  temporary  assumptions  in  A'  Fields  in  £\  do  not  occur  in  A' 

T ;  (A,  A')  hiVj  let  x  =  ex  in  e2  :  E2  \  £\  U  £2 

P-Meth 

(x  :  T,  this  :  C);  P  \~q  e  :  3result  :  Tr.Pr  <S>  T  \  £ 

E  =  3result :  Tr.Pr  override(m,  C,\/x  :  T.P  — o  P) 

Pr  m(T  x)  :  P  — o  P  =  e  ok  in  C* 

P-Class _  _ 

Af  ok  in  (7  A/  overrides  all  methods  with  thisfr  permissions  in  C' 
class  C  extends  C7  {FRINMjok 

P-Prog 

CL  ok  ■■■\-i_e:E\£ 

(CL,  e )  :  E 


Figure  4.6:  Permission  checking  for  expressions  (part  1)  and  declarations  (helper  judgments  in  Figure  4.7) 
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4.2.1  Permission  Tracking 

We  permission-check  an  expression  e  with  the  judgment  F;  A  \~lc  e  :  3x  :  T.P  \  £.  This  is  read  as,  “in 
valid  context  T  and  linear  context  A,  an  expression  e  executed  within  receiver  class  C  has  type  T,  yields 
permissions  P,  and  affects  fields  £”.  Permissions  A  are  consumed  in  the  process.  We  omit  the  receiver  C 
where  it  is  not  required  for  checking  a  particular  syntactic  form.  The  set  £  keeps  track  of  fields  that  were 
assigned  to,  which  is  important  for  the  correct  handling  of  permissions  to  fields.  It  is  omitted  when  empty. 
The  marker  i  in  the  judgment  can  be  0  or  1  where  i  =  1  indicates  that  states  of  objects  in  the  context  could 
change  during  evaluation  of  the  expression.  This  will  help  us  reason  about  temporary  state  assumptions.  A 
combination  of  markers  with  i  V  j  is  I  if  at  least  one  of  the  markers  is  1. 

valid  contexts  T  ::=  •  |  T,x:T  \  T,o:C  \  T,z:H  \  T,q 

linear  contexts  A  ::=  ■  |  A,P 

effects  £  ::=  •  |  £,f 

Valid  and  linear  contexts  distinguish  valid  (so-called  persistent)  facts  (T)  from  resources  (A,  also  called 
ephemeral  facts).  Resources  are  tracked  linearly,  forbidding  their  duplication,  while  valid  facts  can  be  used 
arbitrarily  often.  (In  logical  terms,  contraction  is  defined  for  valid  facts  only).  The  valid  context  types  object 
variables,  fraction  variables,  and  location  types  and  keeps  track  of  terms  q  known  to  be  true  or  false.  Fraction 
variables  are  tracked  in  order  to  handle  fraction  quantification  correctly.  The  linear  context  holds  currently 
available  resource  predicates. 

Fractions  and  fraction  functions  arc  formally  typed  in  Figure  4.4.  Note  that  fraction  function  types 
keep  track  of  exactly  which  nodes  arc  mapped.  Permission  validity  requires  that  the  fraction  function  of  a 
permission  covers  exactly  the  nodes  between  (and  including)  the  permission’s  root  node  and  the  state  space 
root  alive  (Figure  4.4). 

The  judgment  T  F  t  :  T  types  terms  (Figure  4.5)  and  is  completely  standard.  It  includes  the  usual  rule 
for  subsumption  using  nominal  subtyping  induced  by  the  extends  relation.  Term  typing  is  used  in  expression 
checking. 

Our  expression  checking  rules  are  syntax-directed  up  to  reasoning  about  permissions.  Permission  rea¬ 
soning  is  deferred  to  a  separate  judgment  T;  A  P  P  that  uses  the  rules  of  linear  logic  to  prove  the  availability 
of  permissions  P  in  a  given  context.  This  judgment  will  be  discussed  in  Section  4.2.5.  Permission  checking 
rules  for  most  expressions  appear  in  Figure  4.6  and  arc  described  in  turn.  Packing,  method  calls,  and  field 
assignment  arc  discussed  in  following  subsections.  Flelper  judgments  arc  summarized  in  Figure  4.7.  The 
notation  [e' /x\e  substitutes  e'  for  occurrences  of  x  in  e. 

•  P-Term  embeds  terms.  It  formalizes  the  standard  logical  judgment  for  existential  introduction  and 
has  no  effect  on  existing  objects. 

•  P-Field  checks  field  accesses  analogously. 

•  P-New  checks  object  construction.  The  parameters  passed  to  the  constructor  have  to  satisfy  initializa¬ 
tion  predicate  P  and  become  the  object’s  initial  field  values.  The  new  existentially  quantified  object 
is  associated  with  a  unique  permission  to  the  root  state  that  makes  state  assumptions  according  to  the 
declared  start  state  A.  Object  construction  has  no  effect  on  existing  objects. 

The  judgment  init  (Figure  4.7)  looks  up  initialization  predicate  and  start  state  for  a  class.  The  start 
state  is  a  conjunction  of  states  (Figure  4.3).  The  initialization  predicate  must  be  strong  enough  to 
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class  C  extends  C'  {. . .}  E  CL 
C  extends  C' 

class  C  {. . .  M  . . .}  G  CL  Tr  m(T  x)  :  P  — °  Bresult  :  Tr.P'  =  e  G  M 
mtype(m,  C)  =  Vx  :  T.P  — o  Bresult :  Tr.P' 

C  extends  C'  mtype(m,  C1)  =  \/x  :  T.MS'  implies  (x  :  T,  this  :  C)\  •  b  MS  — °  MS' 

override(m,  C,  Vx  :  T.MS ) 

class  C  . . .  {F  . . .}  G  CL 

localFields(C)  =  F  init(Object)  =  (1,  alive) 

class  C  extends  C'  {/  :  T  in  n  S  initially  ( P 1  (g>  P,A)  . . .  } 
init(C'/)  =  (V/'  :  T'.P1,  A')  •;  (P,  access(super,  alive,  {alive  i— >  1 } ,  ) )  b  invc(alive,  A)  <8>  T 

init(C')  =  (VfTP7,  JCT.P'  <g>  P,  A) 

_  p  =  p  redc(n,/) 

class  C  {. . .  n  =  P  .  .  .}  G  CL  n'<n"<n 

predG(n)  =  P  pred  c(n',n)  =  P 

invc(A)  =  P  =>■  n' 

in vG(n,  A)  =  P  ®  predc(n/,  n)  <g>  predG(n)  invG(ro)  =  1  =^>  n 

invc(A:)  =  Pi  =A  m  pred c(rii,  n)  =  P[  m  <g>  ra2  <C  n  (i  =  1,  2) 

invc(Ai  <8>  A2)  =  Pi  <g>  P{  <g>  P2  <g>  P2  =>■  n 

invc(^4i)  =  Pt  =>  rii  predG(rii,  n)  =  P[  iii®n2«n  (*  6  1,2) 

invG(^4i  ©  A2)  =  (Pi  <g>  P{)  ©  (P2  <g>  P2)  =>■  n 

only  pure  permissions  in  P  exists  share,  full,  or  unique  permission  in  P 

effects  A I  lowed  (P)  =  0  effects  A I  lowed  (P)  =  1 


Figure  4.7:  Protocol  verification  helper  judgments 
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prove  the  invariant  needed  for  the  start  state.  invc(n,  A)  constructs  that  invariant  for  root  node  n 
(here,  alive)  and  state  information  A. 

•  P-If  makes  sure  that  the  conditional  term  is  of  Boolean  type  and  then  assumes  its  truth  (falsehood)  in 
checking  the  then  {else)  branch.  This  approach  lets  branches  make  use  of  the  tested  condition.  The 
disjunction  used  for  typing  if  expressions  reflects  that  we  statically  do  not  know  which  branch  will  be 
taken.  Instead,  the  condition  determines  this  choice  at  run-time. 

•  P-Let  checks  a  let  binding  and  represents  existential  elimination,  complementing  P-Term.  The 
linear  context  used  in  checking  the  second  subexpression  must  not  mention  fields  affected  by  the  first 
expression.  This  makes  sure  that  outdated  field  permissions  do  not  “survive”  assignments  or  packing. 
Moreover,  temporary  state  information  is  dropped  if  the  first  subexpression  has  side  effects. 

A  program  consists  of  a  list  of  classes  and  a  main  expression  (P-PROG,  Figure  4.6).  As  usual,  the  class 
table  CL  is  globally  available.  The  main  expression  is  checked  with  initially  empty  contexts.  The  judgment 
CL  ok  (P-CLASS)  checks  a  class  declaration.  It  checks  fields,  states,  and  invariants  for  syntactic  correct¬ 
ness  in  the  obvious  way  (omitted  in  the  rule)  and  verifies  consistency  between  method  specifications  and 
implementations  using  the  judgment  M  ok  in  C.  P-Meth  assumes  the  specified  pre-condition  of  a  method 
(i.e.  the  left-hand  side  of  the  linear  implication)  and  verifies  that  the  method’s  body  expression  produces 
the  declared  post-condition  (i.e.  the  right-hand  side  of  the  implication).  Conjunction  with  T  drops  excess 
permissions,  e.g.,  to  temporary  objects.  The  override  judgment  concisely  enforces  behavioral  subtyping  (see 
Section  4.1.6).  A  method  itself  is  not  a  linear  resource  since  all  resources  it  uses  (including  the  receiver)  are 
passed  in  upon  invocation;  method  declaration  checking  therefore  corresponds  to  universal  introduction. 

4.2.2  Packing  and  Unpacking 

We  use  a  refined  notion  of  unpacking  (DeLine  and  Fahndrich,  2004b)  to  gain  access  to  fields:  we  unpack 
and  pack  a  specific  permission.  Unpacking  means  to  replace  a  permission  for  the  receiver  object  with 
permissions  for  its  fields  implied  by  the  invariant  of  the  receiver’s  current  state;  packing  does  the  opposite. 
The  access  to  fields  we  gain  from  unpacking  reflects  the  permission  we  unpacked:  full  and  share  permissions 
give  modifying  access,  while  a  pure  permission  gives  read-only  access  to  underlying  fields. 

Packing  helps  dealing  with  transitions  between  scopes  (methods),  where  problems  can  occur  when  the 
same  fields  arc  accessed  from  different  scopes,  in  particular  in  the  presence  of  reentrancy.  To  avoid  incon¬ 
sistencies,  objects  arc  always  fully  packed  when  methods  arc  called.  To  simplify  the  situation,  only  one 
permission  can  be  unpacked  at  the  same  time.  Intuitively,  we  “focus”  (Fahndrich  and  DeLine,  2002)  on 
that  permission.  This  lets  us  unpack  share  like  full  permissions,  gaining  full  rather  than  shared  access  to 
underlying  fields  (if  available).  The  syntax  for  packing  and  unpacking  is  as  follows. 

expressions  e  ::=  ...  |  unpack(n,  k,  A)  in  e 

pack  to  A  in  e 

Packing  and  unpacking  always  affects  the  receiver  of  the  currently  executed  method.  The  unpack  pa¬ 
rameters  express  the  programmer’s  expectations  about  the  permission  being  unpacked.  For  simplicity,  an 
explicit  subtree  fraction  k  is  paid  of  unpack  expressions.  It  could  be  inferred  from  a  programmer-provided 
permission  kind,  e.g.  share. 
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\nvc(n,g,k,A) 
\nvc(n,g,0,A) 
where  abovec(n) 


in vc(n,  A)  <g>  purify(abovec(?z)) 
purify  (invc(n,  A)  <g>  above<y(n)) 
^Qn':n<n'<  alive  Pred  c(ri) 


Figure  4.8:  Invariant  construction  (purify  in  Figure  4.10) 


Typechecking.  In  order  for  pack  to  work  properly  we  have  to  “remember”  the  permission  we  unpacked. 
Therefore  we  introduce  unpacked  as  an  additional  linear  predicate. 

permissions  p  ::=  ...  |  unpacked  (n,  g,  k,  A) 

The  checking  rules  for  packing  and  unpacking  are  given  in  Figure  4.9.  Notice  that  packing  and  unpack¬ 
ing  always  affects  permissions  to  thisfr.  (We  ignore  substitution  of  this  with  an  object  location  at  runtime 
here.) 

P-Unpack  first  derives  the  permission  to  be  unpacked.  The  judgment  inv  determines  a  predicate  for 
the  receiver’s  fields  based  on  the  permission  being  unpacked.  It  is  used  when  checking  the  body  expression. 
An  unpacked  predicate  is  added  into  the  linear  context.  We  can  prevent  multiple  permissions  from  being 
unpacked  at  the  same  time  using  a  straightforward  dataflow  analysis  (omitted  here). 

P-Pack  does  the  opposite  of  P-Unpack.  It  derives  the  predicate  necessary  for  packing  the  unpacked 
permission  and  then  assumes  that  permission  in  checking  the  body  expression.  The  new  state  assumption 
A  can  differ  from  before  only  if  a  modifying  permission  was  unpacked.  Finally,  the  rule  ensures  that 
permissions  to  fields  do  not  “survive”  packing. 

Invariant  transformation.  The  judgment  in vc(n,  g,  k,  A)  determines  what  permissions  to  fields  are  im¬ 
plied  by  a  permission  access(thisfr,  n,  g.  k.  A)  for  a  frame  of  class  C.  It  is  defined  in  Figure  4.8  and  uses 
in vc(n,  A)  (which  we  saw  in  Figure  4.7)  to  look  up  declared  invariants  and  a  purify  function  (Figure  4.10) 
to  convert  arbitrary  permissions  into  pure  permissions. 

Unpacking  a  full  or  shared  permission  with  root  node  n  yields  purified  permissions  for  nodes  “above” 
n  and  includes  invariants  following  from  state  assumptions  as-is.  Conversely,  unpacking  a  pure  permission 
yields  completely  purified  permissions. 

4.2.3  Calling  Methods 

Checking  a  method  call  involves  proving  that  the  method’s  pre-condition  is  satisfied.  The  call  can  then  be 
typed  with  the  method’s  post-condition.  The  rules  for  method  calls  perform  universal  elimination,  comple¬ 
menting  P-Meth. 

Unfortunately,  calling  a  method  can  result  in  reentrant  callbacks.  In  order  to  ensure  that  objects  arc 
consistent  when  called  we  require  them  to  be  fully  packed  before  method  calls.  This  reflects  that  aliased 
objects  always  have  to  be  prepared  for  reentrant  callbacks. 

This  is  not  a  limitation  because  we  can  always  pack  to  some  intermediate  state.  Notice  that  such  inter¬ 
mediate  packing  obviates  the  need  for  adoption  while  allowing  focus  (Fahndrich  and  DeLine,  2002):  the 
intermediate  state  represents  the  situation  where  an  adopted  object  was  taken  out  of  the  adopting  object. 
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P-Unpack 

T;  A  \~c  access(thisfr,n,  g,  k,  A) 

receiver  packed  k  =  0  implies  i  =  0  F:  (A7,  in  vc(n,  g,  k,  A),  un packed (n,  g ,  k,  A))  \~lc  e  :  E  \  £ 

r ;  (A,  A;)  V-'q  unpack(n,  k,A)  in  e  :  E  \  £ 

P-Pack 

T;  A  \~c  invc(n,  g,  k ,  il)  <8>  unpacked(n,  <7,  k,  A ') 
k  =  0  implies  A  =  A'  T;  (A',  access (thisfr,  n,  g,  k,  A))  \~lc  e  :  E  \  £ 
localFields(C)  =  f  :  T  in  n  Fields  do  not  occur  in  A' 
r ;  (A,  A')  hp  pack  n  to  A  in  e  :  E  \  f 

P-Call  _ 

T  h  to  :  Cq  r  h  t  :T  T;  A  h  [to/this]  [to/thisfr][i/x]P 

mtype(?n,  Co)  =  Vx  :  T.P  -«  E  i  =  effectsAllowed(P)  receiver  packed 

TjAh*  to-m(t)  :  [fo/this][to/thisfr][f/x]P 

P-SUPER  _ 

T\-t:T  T;  A  h  [super/thisfr]  [t/x]P 

C  extends  C'  mtype(m,  C')  =  Vx  :  T.P  E  i  =  effectsAllowed(P)  receiver  packed 

r;Ah^  super. m(t)  :  [super/thisfr][f/x]P 

P-Assign 

r;  A  F  t  :  3x  :  Tj.P  r;  A'  Fc  [fi/x']P'  ®  p 
localFields(C)  =  /  :  T  in  n  rii  <  n  p  =  unpacked(n,  g,  k,  A),  k  7^  0 

F;  (A,  A')  h  c  assign/,;  :=  t  :  3x'  :  T-.P'  <g>  [fi/x\P®p\  fo 

Figure  4.9:  Permission  checking  for  expressions  (part  2) 


p  =  access(r,  n,g,  k,  A)  purify(Pi)  =  P[  purify(P2)  =  P'2  ope{<g),&,0} 

purify  (p)  =  pur e(r,n,g,A)  purify(Pi  op  P2)  =  P[  op  P*2 

unit  6  {1,  T,  0}  purify (P)  =  P'  purify (P)  =  P' 

purify(unit)  =  unit  purify(zk  :  H.P)  =  3z  :  H.P'  purify (Vz  :  H.P)  =  Yz  :  H.P' 

Figure  4.10:  Permission  purification 
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Virtual  calls.  Virtual  calls  arc  dynamically  dispatched  (rule  P-Call).  In  virtual  calls,  frame  and  object 
permissions  arc  identical  because  object  permissions  simply  refer  to  the  object’s  virtual  frame.  This  is 
achieved  by  substituting  the  given  receiver  for  both  this  and  thisf,. 

Super  calls.  Super  calls  arc  statically  dispatched  (rule  P-Super).  Recall  that  super  is  used  to  identify 
permissions  to  the  super-frame.  We  substitute  super  only  for  thisfr.  We  omit  the  identity  substitution  of  this 
for  the  receiver  (this  again)  for  clarity. 

4.2.4  Field  Assignments 

Assignments  to  fields  change  the  state  of  the  receiver’s  current  frame.  We  point  out  that  assignments  to  a  field 
do  not  change  states  of  objects  referenced  by  the  field.  Therefore  reasoning  about  assignments  mostly  has 
to  be  concerned  with  preserving  invariants  of  the  receiver.  The  unpacked  predicates  introduced  in  Section 
4.2.2  help  us  with  this  task. 

Our  intuition  is  that  assignment  to  a  field  requires  unpacking  the  surrounding  object  to  the  point  where 
all  states  that  refer  to  the  assigned  field  in  their  invariants  arc  revealed.  Notice  that  the  object  does  not  have 
to  be  unpacked  completely  in  this  scheme.  For  simplicity,  each  field  is  annotated  with  the  subtree  that  can 
depend  on  it  (Figure  4.1).  Thus  we  interpret  nodes  as  data  groups  (Leino,  1998). 

The  rule  P-ASSIGN  (Figure  4.9)  assigns  a  given  object  t  to  a  field  /',  and  returns  the  old  field  value  as 
an  existential  x' .  This  preserves  information  about  that  value.  The  rule  verifies  that  the  new  object  is  of  the 
correct  type  and  that  a  suitable  full  or  share  permission  is  currently  unpacked.  By  recording  an  effect  on  ft 
we  ensure  that  information  about  the  old  field  value  cannot  “flow  around”  the  assignment  (which  would  be 
unsound). 

4.2.5  Permission  Reasoning  with  Splitting  and  Joining 

Our  permission  checking  rules  rely  on  proving  a  predicate  P  given  the  current  valid  and  linear-  resources, 
written  T;  A  P  P  (Figure  4.1 1).  We  use  standard  rules  for  the  decidable  multiplicative-additive  fragment  of 
linear-  logic  (MALL)  with  quantifiers  that  only  range  over  fractions  (Lincoln  and  Scedrov,  1994).  Following 
Zhao  (2007)  we  introduce  a  notion  of  substitution  into  the  logic  that  allows  substituting  a  set  of  linear 
resources  with  an  equivalent  one  (Subst,  Figure  4.11),  similar  to  a  conventional  subtyping  rule  in  non¬ 
linear-  logics. 

The  judgment  P  ^  P'  defines  legal  substitutions.  We  use  substitutions  for  splitting  and  merging 
permissions  (Figure  4.12).  The  symbol  <00*  indicates  that  transformations  are  allowed  in  both  directions. 
Sym  and  Asym  generalize  the  rules  from  Section  3.1.  Most  other  rules  are  used  to  split  permissions  for 
larger  subtrees  into  smaller  ones  and  vice  versa.  We  explain  each  rule  in  turn. 

Sym  symmetrically  splits  a  permission  into  two  equivalent  permissions.  Notice  how  fractions  are  split. 
Asym  asymmetrically  splits  a  pure  permission  off  a  given  permission.  Here,  the  subtree  fraction  k  is  un¬ 
touched,  reflecting  the  asymmetric  split.  Both  transformations  can  be  inverted.  We  require  non-contradicting 
state  information  when  merging  permissions. 

F-S  PUT-0  splits  a  full  permission  with  a  conjunctive  state  assumption  into  a  conjunction  of  full  per¬ 
missions.  F-Merge-<0  inverts  F- Split-0  but  requires  the  fraction  on  the  new  root  node  to  be  1.  This 
guarantees  that  no  additional  full  or  shared  permissions  exist  in  the  new  permission’s  subtree.  F-©  splits 
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LinHyp 

f VPKP 

®  I 

TjAxhP!  r;  A2  h  P-2 
r;(Ai,  a2)  b  Pi®p2 

1 1 

r;  - 1- 1 
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r;AhP!  r;AhP2 

r ;  A  h  Pi  &  P2 


SUBST 

r;AbP'  p'^p 
r;  a  h  p 

®  E 

r;AhP!®P2  r;(A',P1,P2)hP 

r;(A,A')hP 
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T;  A  h  Pi  &  P2 
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r ;  A  b  Pi 
T ;  A  b  Pi  ©  P2 

T ;  A  b  P2 

r;  a  b  Pi  ©  p2 


no  0  introduction 

VI 

(T,z  :  P);A  b  P 

T;  A  b  Vz  :  P.P 

3  I 

rb/i:P  r;  A  b  [h/z\P 
r;  A  b  3z  :  H.P 


no  T  elimination 


®  E 

r;AbPi®p2  r;(A',Pi)bP  r;(A',p2)bP 
r;(A,A')bP 


0E 

r;Abo 

r;(A,A')bP 

VE 

T  b  h:H  T;  A  b  Vz  :  H.P 

r;Ab  [h/z\p 
3  E 

T;Ab  3z:H.P  (r,  z  :  H),  (A',  P)  b  P' 
r;(A,A')  b  p' 


Figure  4.1 1:  Affine  logic  for  permission  reasoning 
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and  conjoins  full  permissions  with  a  disjunction  of  state  assumptions.  Since  only  one  of  the  two  state  as¬ 
sumptions  can  be  true  at  a  given  time  we  do  not  need  to  split  fractions. 

F-Down  limits  a  full  permission  to  a  smaller  subtree  by  moving  the  root  node  down  in  the  state  space. 
The  fraction  function  is  appended  with  additional  1  fractions  for  nodes  that  arc  above  the  moved  root. 
Notice  that  this  operation  is  only  allowed  if  the  state  assumptions  of  the  original  permission  are  inside  the 
new  root  node  (in  order  to  ensure  well-formedness  of  the  new  permission).  F-Up  does  the  opposite  but  like 
F-MERGE-(g)  it  requires  the  fraction  on  the  new  root  node  to  be  1.  Similarly,  P-Up  can  be  used  to  weaken  a 
pure  permission  by  moving  its  root  up  in  the  state  space.  Finally,  Forget  allows  a  permission  to  “forget” 
its  state  assumption.  This  rule  is  used  to  drop  temporary  state  assumptions  and  can  be  used  to  match  state 
assumptions  between  permissions  prior  to  applying  rules  such  as  Sym,  Asym,  and  F-Down. 

Our  splitting  and  joining  rules  maintain  a  consistent  set  of  permissions  for  each  object  so  that  no  per¬ 
mission  can  ever  violate  an  assumption  another  permission  makes.  Fractions  of  all  permissions  to  an  object 
sum  up  to  (at  most)  1  for  every  node  in  the  object’s  state  space.  These  rules  encompass  the  needed  kinds  of 
permission  manipulations  that  we  arc  aware  of: 

•  Obtaining  a  needed  kind  of  permission,  such  as  a  pure  from  a  full,  with  symmetric  and  asymmetric 
splitting  and  merging  of  permissions  for  a  given  node  (Sym,  Asym). 

•  Splitting  and  merging  permissions  along  dimensions  (F-Split,  F-Join). 

•  Obtaining  a  permission  with  a  needed  root,  by  moving  roots  up  or  down  (F-Down,  F-Up,  P-Up). 

Forget  could  be  made  to  allow  dropping  some  state  assumptions  while  keeping  others,  but  we  omit 
this  generalization  for  clarity. 

4.2.6  Soundness 

Bierhoff  and  Aldrich  (2007a)  present  a  proof  of  soundness  for  a  fragment  of  the  system  presented  in  this 
dissertation.  The  fragment  does  not  include  inheritance  and  only  supports  permissions  for  objects  as  a  whole. 
State  dimensions  are  omitted  and  specifications  arc  deterministic.  The  fragment  does  include  full,  share,  and 
pure  permissions  with  explicit  fractions  and  temporary  state  information.  Pre-conditions,  post-conditions, 
and  invariants  arc  conjunctions  of  such  permissions. 

Soundness  is  proven  with  the  canonical  progress  and  preservation  theorems.  Progress  means  that  a  well- 
typed  program  is  always  either  already  a  value  or  can  take  an  evaluation  step.  Preservation  shows  that  a 
well-typed  program  always  remains  well-typed  when  it  takes  an  evaluation  step.  “Well-typed”  here  means 
that  permissions  and  typestates  correctly  approximate  runtime  behavior.  Therefore,  these  two  theorems 
together  intuitively  prove  that  well-typed  programs  will  never  violate  protocols  declared  in  the  program 
using  permissions  and  typestates. 

4.2.7  Example 

To  illustrate  how  verification  proceeds.  Figure  4.13  shows  the  fill  method  from  Buf f eredlnputStream 
(Figure  3.12)  written  in  our  core  language.  As  can  be  seen  we  need  an  intermediate  state  reads  and  a 
marker  field  reading  that  indicate  an  ongoing  call  to  the  underlying  stream.  We  also  need  an  additional 
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Sym 

A  =  A'  =  A"  or  [A  =  A!  and  A"  =  n)  or  ( A  =  A"  and  A'  =  n ) 
access(r,  n,  g,  A,  A)  access (r,  n,  g/ 2,  A/2,  ^4')  ©  access (r,  n,  5/2,  k/2,  A") 

Asym 

A  =  A7  =  or  (A  =  A/  mid  A”  =  n)  or  (A  =  A"  and  A1  =  n) 
access(r,  n,  g ,  k ,  A)  access(r,  n,  5/2,  k,  A1)  ©  pure(r,  n,  <7/2,  A") 

F-Split-® 

ni  #  77-2  Ai  -<:  ni  <  n  A2  -<  ri2  <  n  pi  =  full(r,  nj,  {5,  nodes(nj,  n)  1— ►  l}/2,Ai) 

full(r,  n,  5,  Ai  ©  A2)  pi  ©  p2 

F-MERGE-® 

n-i  #  n2  4i  4  ni  <  n  A2  A  n2  <  n  pi  =  full(r,  rij,  {5,  m->l,  nodes(nj,  n)  1— >  l}/2,  Aj) 

pi®p2^  fu\\(r,n,{g,n  i-»  l},^4i  ©  A2) 


^4i  #  ^2 


full(r,  n,  <7,  Ai  ©  A2)  full(r,  n,  g,  Ax)  ©full  (r,n,g,A2) 

F-Down 


A  A  n1  <  n 


full  (r,  n,  <7,  ^4)  ^  full (r,  n/,  {5,  nodes(n/,  n)  1— >  1},  A) 


F-Up 


full (r,  n  ,  { g ,  n 


A  A  n  <n 

1,  nodes(n/,  n)  1— >•  1},  A)  ^  full (r,  n,  {5,  n  1— >•  1},  A) 


P-Up 


n  <  n 


pure(?’,  n,  { <7,  nodes(n/,  n)  1— >  A:},  A)  ^  pure(r,  n  ,  A) 


Forget 


access(r,  n,  A’,  A)  ^  access(r,  n,  A,  n) 


Figure  4.12:  Splitting  and  merging  of  access  permissions 
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state  refinement  to  specify  an  internal  method  replacing  the  while  loop  in  the  original  implementation.  (We 
assume  that  thisf,  permissions  can  be  used  for  calls  to  private  methods.) 

Maybe  surprisingly,  we  have  to  reassign  field  values  after  super. read ()  returns.  The  reason  is  that 
when  calling  super  we  lose  temporary  state  information  for  this.  Assignment  re-establishes  this  information 
and  lets  us  pack  properly  before  calling  doFill  recursively  or  terminating  in  the  cases  of  a  full  buffer  or  a 
depleted  underlying  stream. 

It  turns  out  that  these  re-assignments  are  no?  just  an  inconvenience  caused  by  our  method  but  point  to 
a  real  problem  in  the  Java  standard  library  implementation.  We  could  implement  a  malicious  underlying 
stream  that  calls  back  into  the  “surrounding”  Buff eredlnputStream  object.  This  call  changes  a  field, 
which  causes  the  buffer’s  invariant  on  count  to  permanently  break,  later  on  resulting  in  an  undocumented 
array  bounds  exception  when  trying  to  read  beyond  the  end  of  the  buffer  array. 

Because  fill  operates  on  a  share  permission  our  verification  approach  forces  taking  into  account  possi¬ 
ble  field  changes  through  reentrant  calls  with  other  share  permissions.  (This  is  precisely  what  our  malicious 
stream  does.)  We  could  avoid  field  re-assignments  by  having  read  require  a  full  permission,  thereby  docu¬ 
menting  that  reentrant  (modifying)  calls  arc  not  permitted  for  this  method. 
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class  Buff eredlnputStream  extends  FilterlnputStream  { 
states  ready,  reads  refine  open;  . . . 
states  partial,  complete  refine  filled; 

reads  :=  reading ;  ready  :=  reading  =  false; . . . 

private  boolean  reading;  . . . 

public  int  read()  :  Vfc  :  Fract. . . .  = 
unpack(open,  k) in 

let  r  =  reading  in  if (r  ==  false,  . . .  fill()  ...  ) 

private  bool  fill()  :  Vfc  :  Fract. 
share(f/zwfn  open)  in  depleted  ©  eof  — o 
share(/7zzsfr,  open)  in  available  ©  eof  = 
unpack(open,  A;,  depleted  ©  eof)  in 

assign  count  =  0  in  assign  pos  =  0  in 
assign  reading  =  true  in 
pack  to  reads  in 

let  b  =  super. read ()  in 
unpack(open,  k,  open)  in 

let  r  =  reading  in  assign  reading  =  false  in 
assign  count  =  0  in  assign  pos  =  0  in 
if(r,  if (b  =  -1,  pack  to  eof  in  false, 
pack  to  depleted  in  doFill(b)), 
pack  to  eof  in  false) 

private  bool  doFillfint  b)  :  Vfc  :  Fract. 
share(f/zwfn  open)  in  depleted  ©  partial  — o 
share(f/zwfn  open)  in  partial  ©  complete  = 
unpack(open,  k,  depleted  ©  partial)  in 

let  c  =  count  in  let  buffer  =  buf  in 
assign  buffer [c]  =  b  in  assign  count  =  c  +  1  in 
let  1  =  buff er . length  in 
if(c  +  1  >=  1,  pack  to  complete  in  true, 
assign  reading  =  true  in  pack  to  reads  in 
let  b  =  super. read()  in  unpack(open,  k)  in 

let  r  =  reading  in  assign  reading  =  false  in 
assign  count  =  c  +  1  in  assign  pos  =  0  in 
pack  to  partial  in 

if(r  ==  false  I  I  b  ==  -1,  true,  doFill(b)) 


Figure  4.13:  Fragment  of  Buff  eredlnputStream  from  Figure  3.12  in  core  language 


Chapter  5 

Polymorphic  Permission  Inference 


Fractional  permissions  (Boyland,  2003)  have  recently  received  much  attention  for  sound  static  reasoning 
about  programs  that  rely  on  aliasing.  They  have  been  used  for  avoiding  data  races  with  locks  (Terauchi  and 
Aiken,  2008)  as  well  as  for  verifying  properties  of  multi-threaded  (Bornat  et  al.,  2005;  Beckman  et  ah,  2008; 
Leino  and  Muller,  2009)  and  single-threaded  programs  (this  dissertation),  where  fractional  permissions  arc 
typically  embedded  into  a  substructural  logic. 

The  type  system  for  access  permissions  presented  in  the  previous  chapter  is  highly  non-deterministic  (as 
detailed  below).  Intuitively,  challenges  in  automating  the  type  system  arise  from  permission  splitting:  one 
could  potentially  have  to  split  permissions  an  arbitrary  number  of  times  before  a  predicate  such  as  a  method 
pre-condition  is  provable. 

This  chapter  presents  a  type  inference  algorithm  for  proving  permission-based  assertions  in  a  decidable 
fragment  of  linear  logic  (Girard,  1987;  Lincoln  and  Scedrov,  1994).  Unlike  previous  work,  our  inference 
approach  supports  polymorphism  over  fractions,  i.e.,  universally  and  existentially  quantified  fractions. 

The  permission  inference  algorithm  presented  in  this  chapter  extends  conventional  resource  management 
techniques  for  linear  logic  (Cervesato  et  al.,  2000)  to  additionally  infer  how  fractional  permissions  should 
to  be  split  and  merged  over  time.  This  process  requires  collecting  linear  constraints  over  fraction  variables 
(Section  5.1) — which  is  technically  similar  to  previous  work  by  Terauchi  and  Aiken  (2008) — and  ensuring 
that  these  remain  satisfiable  (Section  5.2). 


5.1  Inference  System 

The  permission  type  system  presented  in  the  previous  section  has  several  sources  of  non-determinism;  in 
other  words,  it  makes  several  kinds  of  “guesses”: 

•  Context  splits,  such  as  in  the  linear  logic  proof  rule  for  0,  “guess”  how  permissions  have  to  be  divided 
in  order  to  satisfy  different  predicates.  Notice  that  context  splits  also  appeal-  in  the  typing  rules  in  order 
to  use  different  resources  to  prove  the  branches  of  a  let  binding. 

•  Multiple  proof  rules  can  apply.  For  example,  the  rules  for  proving  a  disjunction  P\  ©  P-2  “guess” 
whether  Pi  or  P2  can  be  proven. 
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•  Permissions  can  be  split  an  arbitrary  number  of  times  and  merged  back  together.  The  type  system 
“guesses”  exactly  how  a  given  permission  has  to  be  split  up  and  merged  in  order  to  satisfy  different 
predicates.  Notice  that  this  additional  non-determinism  is  not  an  issue  in  conventional  linear  logic 
proof  search  because  resources  are  indivisible.  However,  permission  splits  intuitively  happen  together 
with  context  splits  in  order  to  use  the  split-up  permissions  for  proving  different  predicates. 


This  section  presents  a  type  inference  algorithm  for  inferring  how  permissions  flow  through  a  program. 
This  algorithm  is  based  on  conventional  methods  for  resource  management  in  linear  logic  (Cervesato  et  al., 
2000)  and  adds  linear  constraints,  similar  to  constraint  logic  programming  (Jaffar  and  Lassez,  1987),  in 
order  to  infer  permission  splits  and  merges.  The  constraints  essentially  “delay”  decisions  about  splits  and 
merges  to  make  sure  that  permissions  needed  later  on  in  the  program  can  be  satisfied. 

Our  existing  typechecking  and  proof  judgments  (ignoring  typing  information)  Ah  e  :  3x.P  and  Ah  P 
prove  a  predicate  P  based  on  given  resources  A.  They  essentially  treat  A  as  an  input  and  produce  one 
output,  the  predicate  P  that  could  be  proven. 

In  order  to  manage  context  splits,  resource  management  techniques  for  linear  logic  proof  search  (Cervesato 
et  al.,  2000)  add  an  additional  output,  the  “leftover”  resources  A',  to  these  judgments.  The  idea  of  resource 
management  is  that  we  can  make  context  splits  for  proving  two  predicates  Pi  and  P2  deterministic  by  send¬ 
ing  all  resources  down  the  first  branch  (that  proves  P\ )  and  then  use  the  leftover  resources  to  prove  the 
second  predicate  P>.  such  as  in  the  following  sample  rule  for  proving  multiplicative  conjunction: 

AhPi^A'  A'  h  P2  =>  A" 

A  h  Pi  <g)  P2  =>  A" 

We  extend  this  idea  by  tracking  constraints  C  together  with  the  current  linear  context  A  that  encode  what 
permissions  were  needed  in  the  different  parts  of  the  program.  Again,  we  need  input  and  output  constraints 
so  that  different  parts  of  the  proof  can  add  their  constraints  to  the  existing  ones.  In  fact,  resources  A  will 
always  be  paired  with  their  constraints  C,  usually  written  A  |  C.  We  call  such  a  pair  an  atomic  context. 
Constraints  and  their  collection  will  be  discussed  in  more  detail  below. 

This  leaves  one  final  source  of  non-determinism:  situations  where  multiple  proof  rules  for  linear  logic  arc 
applicable.  This  is  the  case  for  proving  an  external  choice.  Pi  ©  P2  (there  arc  two  right-rules  for  ©  in  Figure 
4.11),  and  for  using  an  internal  choice.  Pi  &  P2  (there  arc  two  left-rules  for  &).  Typically,  backtracking  is 
used  for  these  cases.  Backtracking  means  that  we  arbitrarily  choose  one  of  the  applicable  rules,  and  if  that 
does  not  work,  we  tty  the  other.  But  backtracking  is  awkward  for  our  purposes:  we  are  trying  to  typecheck  an 
entire  program,  which  involves  proving  linear  logic  predicates  for  every  method  call  and  object  construction. 
Whenever  such  a  proof  fails  we  would  have  to  backtrack  to  the  last  call  or  construction  site  in  the  program 
where  we  made  a  choice.  Doing  so  as  paid  of  a  dataflow  analysis  (cf.  Chapter  6),  where  we  are  trying  to 
infer  loop  invariants  as  well,  is  decidedly  non-trivial  because  we  would  have  to  roll  back  the  entire  state 
of  the  flow  analysis,  including  previously  computed  lattice  values  and  nodes  that  still  have  to  be  analyzed. 
Therefore,  instead  of  backtracking,  we  will  carry  all  possible  choices  forward. 

The  following  sections  discuss  the  syntax,  typechecking  rules,  and  proof  rules  of  our  permission  infer¬ 
ence  system. 
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Programs 

A 

• 

1 

A,  m(x i, . . . ,  xn)  :  P  —o  E  =  M 

method 

1 

A,  C(x\,  ...,xn)  :  (P,  A) 

constructor 

Expressions 

M 

x0.m(x  1,  .  ..xn) 

call 

1 

new  C(x i, . . .  ,xn) 

construction 

1 

let  x  =  Mi  in  M2 

sequence 

Expression  types 

E 

::= 

3x.P 

Predicates 

P 

::  = 

access(x,  n,  g)  in  A 

atom 

1 

Pi  <S>  P‘2 

1 

PlkP-2 

1 

Pl®P2 

1 

Vz.P 

1 

3z.P 

Fraction  functions 

9 

::= 

{n\  i— >  k\ , . . . ,  rij  kj,  below  i— >•  k} 

Fractions 

k 

::= 

z 

unknown  fraction 

i 

Z 

fraction  variable 

1 

1 

one 

i 

0 

zero 

State  information 

A 

::= 

77-1 ,  •  •  •  ,  Tlj 

Variables 

r 

::  = 

• 

empty 

1 

r,  x 

variable 

Contexts 

'P 

A  |  C 

atom 

1 

choice 

1 

©  ^2 

all 

Permission  set 

A 

• 

empty 

i 

A  ,P 

extension 

Constraints 

C 

T 

true 

1 

_L 

false 

1 

F 

formula 

i 

CiAC2 

conjunction 

Formulae 

F 

k  =  ki  +  . . .  +  kj 

i 

k\  <  k2 

1 

0  <  k 

Program  variables 

x,y 

;  ;  = 

this  |  ... 

Fraction  variables 

z 

Class  names 

C 

::= 

Object 

Node  names 

n 

::= 

alive 

Method  names 

m 

Figure  5.1:  Syntax  for  permission  inference  system 
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5.1.1  Syntax 

The  syntax  for  the  permission  inference  system  is  summarized  in  Figure  5.1.  Like  the  original  type  system, 
it  is  based  on  Featherweight  Java  (Igarashi  et  ah,  1999),  but  for  simplicity  we  arc  not  keeping  track  of  object 
types.  Instead,  we  are  assuming  that  conventional  typechecking  has  already  ensured  type  safety.  We  arc  also 
ignoring  subtyping  in  this  calculus.  Programs  A  therefore  are  simply  lists  of  methods  (suitably  renamed 
to  avoid  name  clashes)  and  constructors.  Expressions  M  arc  standard;  however,  notice  that  we  introduce  a 
let-binding  construct  to  define  intermediate  variables  and  only  allow  variables  as  arguments  to  methods  and 
object  constructors.  This  lets  programs  written  in  this  language  correspond  more  directly  with  our  original 
type  system. 

Expression  types  existentially  bind  a  variable  that  represents  the  value  computed  by  an  expression 
and  simply  consist  of  a  linear  logic  predicate  P.  Predicates  arc  completely  standard  and  include  ex¬ 
istential  and  universal  quantification  of  fraction  variables  as  in  our  original  type  system.  Permissions 
access).!',  n,  g )  in  A  represent  the  only  kind  of  atomic  predicate.  A  fraction  function  g  for  a  permission 
access(.T, nj, g )  is  written  {n\  >  Aq, . . . ,  n*  i— >  ky,  below  e- >  k},  where  (implicitly)  rii  <  . . .  <  n\  =  alive. 

(Notice  that  the  permission’s  root  matches  the  last  node  entry  in  the  fraction  function.)  It  maps  nodes  n 
to  to  node  fractions',  notice  that  for  readability  we  include  the  below  fraction  in  g.  Fractions  will  include 
unknown  fractions  z  as  well  as  fraction  variables  Z  which  arc  used  during  proof  search  to  find  a  suitable 
instantiation  of  quantified  variables. 

Proof  contexts  T  arc  built  up  from  atomic  contexts  A  |  C  as  described  in  the  previous  section.  Whenever 
we  can  make  a  choice  between  two  rules  we  introduce  a  choice  context  T|  &  T9  that  carries  the  two 
possibilities  forward  (to  avoid  backtracking).  When  one  of  the  choices  fails  to  prove  a  predicate,  we  will 
simply  drop  it,  and  if  no  context  remains  then  our  proof  search  fails,  meaning  there  is  a  protocol  violation 
in  the  program.  It  ends  up  being  convenient  to  also  introduce  all  contexts  T 1  ®  T  2  l<|  encode  that  two 
possibilities  have  to  be  taken  into  account.  This  is  useful  for  situations  where  the  same  resources  need  to 
be  used  in  two  different  subsequent  proofs.  Linear  contexts  A  arc  lists  of  predicates,  and  constraints  C 
arc  conjunctions  of  linear  formulae  F  over  fractions.  We  will  encounter  three  kinds  of  formulae,  namely 
equating  a  fraction  to  a  sum  of  fractions,  relating  two  fractions,  and  asserting  that  a  fraction  is  strictly  greater 
than  zero.  T  will  be  used  for  trivially  true  constraints,  and  we  will  use  _L  for  unsatisfiable  constraints. 

5.1.2  Typechecking  Rules 

Expressions  arc  typechecked  with  the  following  judgment: 

r  |  $hM:  3x.P  =A  'P/ 

This  judgment  can  be  read  as,  in  context  T,  expression  M  (with  free  variables  defined  in  F)  will  produce 
a  value  x  with  predicate  P  as  well  as  a  new  context  T'  for  the  remainder  of  the  program.  The  typechecking 
rules  arc  shown  in  Figure  5.2.  They  rely  on  helper  judgments  shown  in  Figure  5.3.  They  also  use  the 
proof  judgment  $  b  P  =>  $'  which  will  be  discussed  in  the  following  section.  The  rules  for  typechecking 
expressions  arc  in  fact  straightforward  since  constraint  collection  happens  in  the  proof  judgment. 

•  As  before,  T-Call  looks  up  the  invoked  method’s  declared  signature  and  renames  the  parameters, 
including  the  receiver,  to  match  the  arguments  provided.  It  then  proves  the  method  pre-condition  in 
the  given  context,  which  will  produce  a  new  context  that  likely  contains  additional  constraints  in  order 
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T-Call 

mtype(m)  =  Vxo,  xi, . . . ,  xn-P  -<>  E  $  h  P  4 
T  |  'P  F  xo-m(xi, . . . ,  xn)  :  E  ^  T>' 

T-New 

init(C)  =  (Vxi, . .  .,xn.P,A) 

^hP^t7  E  =  3x.access(x,  alive,  {alive  i— >  1,  below  i— r  1})  in  A 
T  |  'P  F  new  C(x i, . . . ,  xn)  :  E  'I'7 

T-Let 

F  |  'P  F  M,  :  3x.P  =>  T7  r,x|f',PbM2:P^  <P" 
r  |  \P  F  let  x  =  Mi  in  M2  :  E  =$■  \ If" 


Figure  5.2:  Typechecking  rules  for  permission  inference 


to  ensure  the  pre-condition’s  satisfiability.  The  new  context  is  used  as  the  output  context  of  the  method 
invocation  expression,  while  the  method  post-condition  types  the  expression. 

•  Similarly,  T-New  creates  a  new  object  of  class  C  by  looking  up  its  initialization  predicate  and  proving 
it  with  the  given  parameters.  The  expression  is  typed  with  the  new  object’s  unique  permission  and 
returns  the  context  resulting  from  proving  the  initialization  predicate. 

•  T-Let  types  let-bindings  by  first  collecting  constraints  from  the  first  branch  and  then  checking  the 
second  branch  using  the  outputs  from  the  first  branch. 

All  typing  judgments  must  be  well-formed  in  order  to  avoid  free  variables  in  permission  predicates. 

TFT'  T  F  M  TFP  TFT'' 
r|f  FM:£^$' 

This  rule  stipulates  that  the  outputs  E  and  \P'  must  only  contain  free  variables  defined  in  T.  In  partic¬ 
ular,  this  means  that  the  variable  defined  in  a  let-binding  cannot  occur  free  in  the  outputs  generated  when 
typechecking  the  binding  expression. 

We  omit  typing  rules  for  field  accesses  and  the  related  packing  and  unpacking  operations  since  they  arc 
orthogonal  to  the  problem  of  permission  inference.  Section  6.3.7  will  discuss  how  these  can  be  handled  in  a 
tool. 

The  constraints  collected  by  the  typechecking  and  proof  rules  arc  similar  in  nature  to  previous  work  on 
inferring  fractions  to  guarantee  data  race  freedom  (Terauchi  and  Aiken,  2008;  Terauchi,  2008),  although  our 
rules  are  complicated  by  fraction  quantification,  the  treatment  of  linear  logic,  and  the  presence  of  fraction 
functions  (instead  of  an  individual  fraction  per  object).  Furthermore,  we  treat  control  flow  differently,  as 
discussed  in  Chapter  6.  Handling  universally  quantified  fractions  seems  to  improve  modularity,  since  each 
caller  of  a  method  can  use  fractions  of  their  choosing  to  instantiate  universal  quantifiers. 
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m(x i, . . . ,  xn)  :  P  — o  E  =  M  6  A 
mtype(m)  =  Ythis,  x\, . . . ,  xn.P  E 


C(x xn)  :  (P,  A)  G  A 
init(C')  =  (Vxi, . .  ,,xn.P,A) 


Figure  5.3:  Helper  judgments  for  permission  inference 


Helper  Judgments.  Helper  judgments  (Figure  5.3)  arc  needed  to  look  up  method  signatures  and  object 
initialization  predicates  and  arc  defined  analogously  to  the  type  system. 

5.1.3  Proof  Rules 

The  judgment  for  proving  a  predicate  P  from  an  atomic  context  is  the  following: 

These  will  be  presented  in  two  steps,  for  deriving  constraints  for  an  individual  permission  and  for  com¬ 
bining  them  when  proving  linear  logic  formulae,  given  an  atomic  context  A  |  C.  Additionally  we  will  have 
context  rules  that  break  down  contexts  until  proof  rules  can  be  invoked  on  atomic  contexts  A  |  C. 


Atom  Rules 

Figure  5.4  shows  the  complete  set  of  rules  for  proving  a  single  permission  in  a  given  context.  In  particular, 
these  rules  handle  permissions  with  root  nodes  coming  from  a  hierarchical  state  space.  We  do  not  model 
state  dimensions  here,  but  it  is  possible  to  extend  Atom-Move-Down  and  Atom-Move-Up  to  multiple 
dimensions,  and  this  is  in  fact  what  our  tool  does  (cf.  Chapter  6). 

There  arc  three  rules  for  splitting  permissions  and  only  one  for  merging  permissions.  The  basic  idea 
behind  these  rules  is  to  delay  root  moves  until  they  are  needed  for  splitting  off  a  given  permission,  while 
permissions  for  the  same  root  arc  merged  eagerly.  A  key  invariant  for  these  rules  to  work  is  that  permissions 
for  the  same  object  but  different  related  roots  ri\  and  ri2 ,  where  m  <  ri2,  can  never  occur  in  this  system.1 

As  mentioned  in  Section  5.1.1,  constraint  formulae  have  three  forms  (Figure  5.1):  they  equate  a  fraction 
sum  with  another  fraction,  define  a  fraction  to  be  strictly  greater  than  0,  or  relate  two  fractions.  The  first 
kind  of  formula  is  used  to  relate  corresponding  fractions  from  different  fraction  functions.  The  other  two 
are  used  to  define  well-formedness  of  fraction  functions  as  explained  below. 

In  order  for  a  fraction  function  {m  i— ►  k\. ... .  nt  i— >  Ay,  below  i— >  k}  to  be  well-formed,  its  fractions 
have  to  be  monotonically  increasing  from  the  fraction  k\  for  the  state  space  root  n \  =  alive  to  ki,  the 
fraction  for  the  permission’s  root  node  rq.  We  achieve  this  with  constraints  Ay  <  . . .  <  Ay.  (Notice  that 
while  ri\  <  ri2  compares  nodes  in  the  state  space,  Ay  <  Ay  compares  fractions.)  This  means  that  nodes 
“higher  up”  in  the  hierarchy  are  mapped  to  a  smaller  fraction  than  nodes  that  refine  them.  This  is  because 
of  root  moves:  fractions  for  “higher  up”  nodes,  in  particular  the  root  node,  may  be  affected  by  permission 

1  They  would  have  to  cover  orthogonal  dimensions,  and  our  implementation  keeps  those  separate  until  they  have  to  be  merged 
into  a  common  root  node,  which  is  achieved  by  generalizing  Atom-Move-Up.  Atom-Move-Down  also  has  to  be  generalized 
to  possibly  place  multiple  leftover  permissions  into  the  output  context,  one  for  each  dimension  materialized  by  the  root  move. 
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Atom-Root-Match 

A  <  A'  g  =  {ni  i— ►  k\ , . . . ,  rij  *— ►  kj,  below  k} 
g  =  {n\  i— >  k\ , i— ►  kj,  below  i— >■  A/}  r/'  =  {jzx  i— >  Zl[, . . .  ,rij  Z"j,  below  i— >  Z"} 

Zl{, . . . ,  Z”,  Z"  fresh  '  C  =  kx  =  k\  +  Z"  A  •  •  •  A  kj  =  k'j  +  Z"  A  k  =  k'  +  Z" 

C"  =  0  <  jfei  <  •  •  •  <  kj  A  0  <  fci  <  •  •  •  <  k'j  A  Z'/  <  •  •  •  <  Z" 

A,  access(x,  rij,  g)  in  A  |  C  P  access(x,  rij,g)  in  A'  A,  access(x,  rij,g")  in  A'  \  C  A  C'  A  C" 


Atom-Move-Down 


<  rij  A  <  A1 

g  =  {ni  h- ►  fci , . . . ,  rij  t— ►  kj,  below  i— >•  k}  g  =  {n\  k[ , . . . ,  rij  i— ►  k'j , . . . ,  nt  k[,  below  i— ►  k'} 

g"  =  {m  •-».  Z", . . . ,  rij  ^  Z", . . . ,  m  ^  Z",  below  ^  Z"}  Z'{,  ■■■  ,  Z",  Z  "fresh 

C'  =  ki  =  k[  +  Z'[  A  ■  ■  ■  A  kj  =  k'j  +  Z"  A  k  =  1  A  1  =  k'j+1  +  Z"+1  A  ■  ■  ■  A  1  =  k[  +  Z"  A  1  =  k'  +  Z" 
c"  =  0  <  ki  <  ■  ■  ■  <  kj  A  0  <  k[  <  ■  ■  ■  <  k[  A  Z'[  <  ■  ■  ■  <  Z" 


A,  access(x,  rij ,  g)  in  A  \  C  P  access(x,  rii,g')  in  A'  A,  access(x,  rii,  g")  in  A!  \  C  A  C'  A  C" 


Atom-Move-Up 

rij  <  rii  A  <  A'  g  =  {n\  *—>  ki , . . .  ,rij  *—>  ki, . . . ,  rij  e- >  kj,  below  i— >  k} 
g  =  {n\  k[, . . .  ,rii  *—>  k'j,  below  i— >■  k'}  g"  =  {n\  i— >  Z![, . . . ,  rij  *—>  Z",  below  i— >  Z"} 

Z’[,  ■  ■  ■  ,  Z",Z"  fresh  C  =  kx  =  k\  +  Z'/  A  ■  ■  ■  A  k%  =  k'j  +  Z"  A  h  =  1 
C"  =  0  <  kx  <  ■  ■  ■  <  kj  A  0  <  k[  <  ■  ■  ■  <  k’j  A  Zl{  <  ■  ■  ■  <  Z" 

A,  access(x,  rij,  g)  in  A  \  C  h  access(x,  rij,  g)  in  A1  A,  access(x,  rij,  g")  in  Af  \  C  A  C'  A  C" 


Atom-Fail 

no  permission  for  x  in  A  that  implies  A 
A  |  C  h  access(x,  rij,  g)  in  A  A  |  C  A  A 


Merge-Match 

A,  access(x,  rij,  g")  C  A  C'  A  C"  h  P  A  l1  g  =  {ni  i— >  fci , . . . ,  rij  i— >  kj,  below  k} 
(j  =  {ni  i — >  A: j , . . . ,  rij  i— >  k'j ,  below  k'}  g"  =  {rai  i— ►  Z '[, . . .  ,rij  i— >  Z'- ,  below  Z7'} 
Z", . . . ,  Z",  Zf '  fresh  '  C'  =  Z"  =  fei  +  &1  A  ■  ■  ■  A  Z"  =  %  +  k'j  A  Z"  =  fc  +  k' 

C"  =  ki  <  ■  ■  ■  <  kj  A  k[  <  ■  ■  ■  <  k'j  A  Z "<•••<  Z" 

A,  access(x,  rij,  g),access(x,  rij,  g')  \  C  h  P  =>■  ^ 


Figure  5.4:  Proof  rules  for  deriving  constraints  for  atomic  permission  predicates 
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splits  that  precede  a  root  “down”  move.  Moving  the  root  down  causes  fractions  for  new  nodes  to  be  added 
to  the  fraction  function,  and  those  fractions  arc  initially  always  1  (see  Figure  4.12). 

But  this  well-formedness  condition  still  permits  node  fractions  in  the  fraction  function  to  be  0,  which  is 
not  legal  for  a  “real”  permission.  Unlike  the  below  fraction,  which  is  0  for  pure  permissions,  node  fractions 
can  never  be  0  according  to  our  splitting  rules  (Figure  4. 12).  Intuitively,  this  is  because  the  node  fractions 
track  how  often  permissions  were  split. 

We  force  a  permission  to  be  real  by  adding  a  constraint  0  <  k\,  where  k\  is  the  fraction  that  the  root 
node  ni  =  alive  is  mapped  to,  which  forces  all  node  fractions  to  be  strictly  positive.  Only  permissions  that 
arc  actually  used  for  proving  a  predicate  have  to  be  real.  Possibly-zero  node  fractions  in  a  fraction  function 
essentially  represent  slack  permissions  at  the  end  of  permission  inference  that  can,  but  do  not  have  to,  be 
real  (and  often  they  are  not  because  other  constraints  force  their  node  fractions  to  be  zero). 

Let  us  see  how  these  constraints  arc  generated  by  our  atomic  proof  rules. 

•  Atom-Root-Match  proves  a  permission  for  some  root  node  n?  if  there  is  a  permission  with  that 
same  root  node  in  the  context.  In  this  case  we  can  add  a  “fresh”  permission  (a  permission  with  a  fresh 
fraction  function)  for  the  same  root  into  the  output  context,  which  represents  the  “leftover”  permission 
after  splitting  off  the  needed  from  the  available  permission.  We  introduce  constraints  that  force  the 
fractions  in  the  needed  and  leftover  permissions  to  sum  up  to  the  fractions  of  the  given  permission. 
We  also  introduce  constraints  for  enforcing  the  well-formedness  of  our  fraction  functions.  Notice  that 
the  leftover  permission  is  not  required  to  be  “real”. 

•  Atom-Move-Down  proves  a  permission  with  a  smaller  root  n,  than  the  one  available  in  the  context 
(rij).  We  again  add  a  fresh  permission  with  the  smaller  root  into  the  leftover  context.  We  introduce 
constraints  that  relate  the  node  fractions  for  nodes  defined  in  the  available  permission  in  the  same  way 
as  before,  but  in  accordance  with  the  original  rule  for  moving  a  root  down  we  require  the  existing 
“below”  fraction  to  be  equal  to  the  literal  1.  And  since  we  are  intuitively  “filling  in“  fractions  for  the 
newly  materialized  nodes,  the  needed  and  leftover  fractions  for  these  new  nodes  have  to  sum  up  to 
1  as  well.  Finally,  we  again  introduce  our  well-formedness  constraints  and  make  the  available  and 
needed  permissions  real. 

•  Atom-Move-Up  does  the  opposite  of  the  previous  rule,  although  it  introduces  slightly  stronger  con¬ 
straints.  In  this  case,  the  available  permission  has  a  smaller  root  than  the  needed  permission,  so  we 
have  to  move  the  root  up.  We  again  introduce  a  fresh  permission  with  the  new  root  and  relate  the  frac¬ 
tions  for  nodes  defined  in  all  three  permissions.  In  accordance  with  the  original  rule  for  moving  up  a 
root  (Figure  4.12)  we  also  require  the  available  fraction  for  the  new  root  to  be  equal  to  1.  Finally,  we 
require  permissions  to  be  well-formed  and  the  available  and  needed  permissions  to  be  real  as  before. 

•  Atom-Fail  applies  when  none  of  the  above  rules  for  proving  an  atomic  permission  apply.  This 
is  either  because  there  arc  no  permissions  for  the  specified  variable,  x,  in  the  context,  or  because  the 
available  permissions  have  insufficient  state  information.  In  either  case,  this  rule  makes  the  constraints 
trivially  unsatisfiable. 

•  Merge-Match  eagerly  merges  two  permissions  in  the  context  with  the  same  root  into  one  and  uses 
that  new  permission  to  prove  the  given  predicate  P  (first  premise).  This  ensures  that  we  always  have 
the  strongest  possible  permission  available  to  prove  the  next  one.  Two  permissions  with  the  same 
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(8)  R  ®L 
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A,  Pi  ©  P2  |  C  b  P  =>-  ^i  © 
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VL 
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A,  P'  |  (7  b  P  =►  A'  |  C' 

A,3z.Pr  |  Cb  P  ^  A'  |  (7' 


Figure  5.5:  Proof  rules  for  linear  logic  formulae 


root  occur  when  additional  permissions  arc  injected  into  the  context  from  method  post-conditions  (cf. 
T-Call  in  Figure  5.2). 


The  constraints  generated  for  moving  a  root  up  arc  slightly  stronger  than  the  ones  for  moving  a  root 
down.  This  reflects  the  asymmetric  nature  of  root  moves:  once  a  root  was  moved  down,  one  cannot  neces¬ 
sarily  move  it  back  up.  (This  is  due  to  additional  dimensions  introduced  in  subclasses.)  This  has  practical 
implications  which  we  address  in  Section  6.4.2.  But  it  is  also,  crucially,  the  reason  for  our  lazy  splitting 
scheme,  which  only  moves  a  root  down  (and  up)  when  we  have  to. 


Linear  Connectives 

Figure  5.5  summarizes  the  judgments  for  proving  linear  logic  formulae  other  than  atomic  permissions.  The 
rules  for  proving  1  i  near  logic  connectives  (<8>,  &,  ©)  are  a  straightforward  adaptation  of  resource  management 
for  lineal-  logic.  In  particular,  constraints  are  simply  threaded  through  the  proof  in  the  obvious  way.  Notice 
that  we  use  our  “choice”  and  “all”  contexts  for  handling  connectives  other  than  <g)  to  avoid  backtracking  (see 
Section  5.1.1).  Furthermore,  we  use  conventional  strategies  for  inferring  quantifier  instantiations.  Notice 
that  the  rules  for  &  and  ©  as  well  as  the  rules  for  V  and  3  are  exactly  dual  to  each  other,  as  they  should  be. 
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Ctx-  & 

I”  P  => 
I”  P  =>  &  ^2 


Ctx-© 

1-  P  =►  ^2 

®  T>2  I"  P  =*>  ©  'P'2 


Figure  5.6:  Context  proof  rules 


Context  Rules 

Finally,  the  rules  for  breaking  down  choice  and  all  contexts  in  order  to  invoke  the  linear  logic  proof  judgment 
presented  in  the  preceding  sections  are  shown  in  Figure  5.6.  The  rules  stipulate  that  predicates  arc  proven 
separately  for  compound  contexts  and  put  back  together  into  a  new  compound  context  afterwards. 

Focusing 

The  rules  as  presented  arc  still  not  fully  deterministic:  we  can  choose  to  break  down  a  premise  (and  which 
premise)  or  to  break  down  the  conclusion.  The  order  in  which  premises  and  conclusions  arc  broken  down 
is  relevant,  for  instance,  for  proving  ,4  &  B  F  ,4  &  B  (if  the  premise  is  broken  down  before  the  conclusion 
then  the  proof  will  fail  while  breaking  down  the  conclusion  first  will  succeed). 

Since  there  arc  only  finitely  many  orders  in  which  one  could  break  down  premises  and  conclusions 
(because  they  never  get  bigger),  backtracking  can  be  used  to  find  an  order  that  allows  proving  the  needed 
conclusion.  Focusing  is  the  standard  technique  for  making  the  choices  for  breaking  down  premises  and 
conclusions  deterministic  (Andreoli,  1992).  Since  the  formalization  of  focusing  for  linear  logic  is  lengthy 
and  standard  we  omit  it  here. 

5.1.4  Soundness  and  Completeness 

In  the  original  presentation  of  permission  splitting  and  merging  we  divided  fractions  (and  fraction  functions) 
in  half  and  put  them  back  together,  using  the  intuition  that  /  =  //2  +  //2.  This  was  mostly  done  for 
notational  convenience:  it  seems  intuitive  to  think  of  splitting  and  merging  as  equating  fractions  /  =  g  +  h, 
but  that  would  have  required  us  to  generate  fresh  fraction  names  everywhere,  which  we  avoid  in  Chapter  4 
for  notational  clarity. 

This  notational  difference  makes  it  hard  to  establish  a  direct  correspondence  between  the  two  systems. 
It  certainly  appeal's  that  if  there  is  a  typing  derivation  in  the  original  system  then  there  is  a  derivation  in  the 
inference  system  with  satisfiable  constraints.  The  converse  is  less  obvious.  Nonetheless  we  believe  that  our 
inference  algorithm  is  faithful  to  the  original  system  and  preserves  safety,  i.e.,  progress  and  preservation, 
if  a  derivation  with  satisfiable  constraints  can  be  found.  Constraint  satisfiability  will  be  discussed  in  the 
following  section. 

5.2  Solving  Constraints 

The  collection  of  constraints  during  typechecking  allows  us  to  effectively  accumulate  information  about  all 
the  permissions  needed  in  a  piece  of  code.  But  a  program  should  only  typecheck  if  constraints  are  satisfiable. 
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T-Meth 

this,  xi, . . . ,  xn  |  P  F  M  :  E  =>■  \F  |= 
m(x i, . . . ,  xn)  :  P  E  =  M  6  A 

Sat-Atom  S  AT-&-L  Sat-&-R  Sat-® 

|=  C  |=  ’J'i  |=  ^2  |=  'I7!  |=  ^2 

|=  A  |  C  |=  &  ^2  |=  &  ^2  |=  ^1  ©  ^2 

Figure  5.7:  Well-defined  methods 

5.2.1  Checking  Method  Definitions 

We  could  check  satisfiability,  written  |=  C,  after  typechecking  every  program  expression.  But  there  is  no 
point  since  constraint  satisfiability  for  the  surrounding  expression  implies  that  the  constraints  for  its  sub¬ 
expressions  arc  satisfied  (because  constraints  arc  conjunctions  that  only  ever  grow  in  length).  Therefore, 
we  can  delay  constraint  resolution  to  the  outermost  expression,  which  in  our  case  are  the  method  bodies. 
The  rule  for  ensuring  that  a  method  body  is  well-defined  is  shown  in  Figure  5.7.  The  rules  for  context 
satisfiability  require  constraint  satisfiability  for  atomic  contexts  and  let  us  choose  a  component  context  in 
“choice”  contexts,  while  both  components  must  be  satisfiable  in  “all”  contexts. 

5.2.2  Constraint  Satisfiability 

Constraints  contain  “unknown”  variables,  written  z,  and  logic  variables  Z.  The  quantifier  proof  rules  of 
Figure  5.5  introduce  both  kinds  of  variables,  while  the  atomic  predicate  rules  (Figure  5.4)  only  introduce 
logic  variables.  Logic  variables  can  be  instantiated  arbitrarily  to  satisfy  constraints.  Unknown  variables, 
on  the  other  hand,  are  assumed  to  have  a  fixed  but  unknown  value.  Nonetheless,  we  arc  allowed  to  make 
certain  assumptions  about  them,  which  come  from  constraints  for  well-formedness  and  permissions  being 
“real”.  In  other  words,  assumptions  about  unknowns  arc  formulae  0  <  z\  <  <  z%.  Assumptions  can 

never  contain  _L. 

In  the  interest  of  clarity  we  will  explicitly  quantify  logic  and  unknown  variables  with  existential  and 
universal  quantifiers,  respectively.  These  quantifiers  correspond  to  our  intuition  that  logic  variables  can  be 
instantiated  with  a  particular  value  while  unknowns  can  represent  any  fraction.  Constraints  therefore  define 
the  following  logical  formula: 

(5.1)  Vzi, . . . ,  zn.(C\  — >  (3Zi, . . . ,  Zm.C2)) 

Here,  C\  is  a  conjunction  of  assumptions  as  described  above  and  C 2  is  an  arbitrary  constraint  formula 
(see  the  constraint  syntax  in  Figure  5.1).  The  Zi  arc  explicitly  quantified  unknown  variables  and  the  Z;  arc 
explicitly  quantified  logic  variables. 

Constraints  of  this  form  (that  arc  not  trivially  true  or  false)  can  be  checked  for  satisfiability  using  Fourier- 
Motzkin  elimination.2  In  a  nutshell,  Fourier-Motzkin  allows  eliminating  a  given  variable  from  a  conjunction 
of  lineal-  constraints  by  re-writing  the  input  formula  into  an  equivalent  formula  that  does  not  include  the 

2Linear  programming  (Schrijver,  1998)  can  unfortunately  not  deal  with  altematingly  quantified  variables  directly. 
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eliminated  variable  (Schrijver,  1998).  In  particular,  the  input  formula  is  satisfiable  if  and  only  if  the  new 
formula  is.  We  can  successively  eliminate  all  variables  from  our  constraints  using  Fourier-Motzkin,  which 
will  result  in  a  final  constraint  formula  that  is  equivalent  to  the  input  formula  and  contains  only  rational 
constants  but  no  variables.  Equality  and  inequality  between  rational  constants  (such  as  0  <  1  or  1  =  1), 
however,  is  decidable. 

In  order  to  deal  with  the  alternating  quantification  in  (5.1)  we  use  Fourier-Motzkin  to  first  eliminate 
the  existentially  quantified  variables,  then  rewrite  the  outer  universal  quantifier  into  an  existential  quantifier 
(which  introduces  negations),  and  finally  use  Fourier-Motzkin  to  eliminate  the  remaining  and  now  existen¬ 
tially  quantified  variables.  This  proceeds  as  follows. 

1.  We  eliminate  the  inner,  existential  quantifier  (the  Z\, ... ,  Zm  in  equation  (5.1)  above).  This  yields  a 
equivalent  but  only  universally  quantified  formula: 

\/zi,  ...  ,  Zn.(C\  - >  C2) 

2.  We  replace  all  equality  constraints  of  the  form  k  =  k\  +  . . .  +  kn  in  C2  with  the  (equivalent)  conjunc¬ 
tion  of  relational  constraints  (k  <  ki  +  ...  +  kn )  A  (k  >  k\  +  . . .  +  Ay,  ),  yielding  another  equivalent 
formula: 

Vzi, ,  zn.{C\  — >  C2) 

3.  Rewrite  the  formula  with  an  existential  quantifier  and  desugar  the  implication  in  the  usual  way: 

^(3z1,...,zn.^C1  VC")) 

Which  simplifies  to 

-l(3zi,  •  •  • ,  zn-C\  A  ~[C2) 

4.  Since  C2  =  F\  A  ...  A  F/  is  a  conjunction  of  relational  constraints,  we  can  trivially  negate  each 
conjunct  iy,  yielding  new  relational  constraints  F'.  Applying  DeMorgan’s  rules  yields 

-l(3zi,  ■  •  ■  ,zn.  \J  Ci  A  F[) 

iei,...,i 

5.  Finally,  we  push  the  existential  quantifier  inwards 

(5.2)  -(  \/  •  •  -iZn.Ci  A  F[) 

6.  We  can  apply  Fourier-Motzkin  (or  linear  programming)  to  decide  the  satisfiability  of  each  disjunct. 
The  original  constraints  (5.1)  arc  satisfiable  if  and  only  if  all  disjuncts  in  (5.2)  are  rmsatisliablc. 

Thus,  Fourier-Motzkin  elimination  can  be  used  to  check  our  fraction  constraint  formulae  (5.1)  for  sat¬ 
isfiability.  Nipkow  (2008)  discusses  more  sophisticated  algorithms  for  eliminating  linear  quantifiers,  which 
arc  applicable  to  our  constraint  formulae.  We  leave  the  evaluation  of  these  algorithms  to  future  work. 

As  it  turns  out,  the  order  in  which  variables  arc  eliminated  in  the  first  step  affects  the  length  of  the  result¬ 
ing  formula.  Shorter  formulae  result  from  eliminating  variables  from  right  to  left  in  relational  constraints 
Zi  <  ...  <  Z„  because  that  minimizes  the  number  of  other  variables  that  the  currently  eliminated  variable 
is  related  to. 


Chapter  6 

Plural:  Java  Tooling 


Our  prototype  tool.  Plural1  (“Permissions  Let  Us  Reason  about  ALiasing”),  is  a  plug-in  to  the  Eclipse  IDE 
that  embodies  the  permission  inference  approach  presented  in  the  preceding  chapter  as  a  static  dataflow  anal¬ 
ysis  for  Java.  In  the  remainder  of  this  chapter  we  show  example  annotations  and  explain  how  permissions 
arc  tracked  and  API  implementations  arc  verified.  Then  we  discuss  tool  features  we  found  useful  in  practice 
and  for  dealing  with  a  real  programming  language.  Notice  that  Plural  assumes  the  analyzed  program  to  be 
properly  synchronized;  possible  thread  interleavings  over  the  analyzed  program  arc  not  considered. 

Nels  Beckman  helped  with  implementing  the  Plural  tool  and  in  particular  contributed  to  API  implemen¬ 
tation  checking  (Section  6.3.7)  and  tracking  concrete  predicates  (Section  6.4.6).  He  also  implemented  an 
extension  to  Plural,  called  NIMBY,  that  checks  protocol  compliance  of  concurrent  software  using  transac¬ 
tional  memory  for  synchronization  (Beckman  et  al.,  2008). 


6.1  Developer  Annotations 

Developers  use  Java  5  annotations  to  specify  method  pre-  and  post-conditions  with  access  permissions.  Fig¬ 
ure  6.1  shows  a  simplified  ResultSet  protocol  with  Plural’s  annotations  (recall  Figure  1.1  and  the  discus¬ 
sion  of  JDBC  protocols  in  Chapter  1).  Annotations  on  methods  (parameters)  specify  borrowed  permissions 
for  the  receiver  (resp.  the  annotated  parameter).  Borrowed  permissions  arc  returned  to  the  caller  when  the 
method  returns.  The  attribute  “guarantee”  specifies  a  state  that  cannot  be  left  while  the  method  executes. 
For  example,  next  advances  to  the  next  row  in  the  query  result,  guaranteeing  the  result  set  to  remain  open. 
Conversely,  a  required  (ensured)  state  only  has  to  hold  when  the  method  is  called  (returns).  For  instance, 
only  after  calling  getlnt  is  it  legal  to  call  wasNull. 

Annotations  are  therefore  the  main  way  in  which  developers  interact  with  Plural.  Major  annotations 
include  the  following. 

•  ©Unique,  ©Full,  ©Share,  ©I mm,  and  ©Pure  declare  unique,  full,  share,  immutable,  and  pure  per¬ 
missions,  respectively,  for  the  annotated  method  parameter  or,  if  they  arc  placed  on  a  method,  the 
method’s  receiver2.  They  define  the  following  attributes: 

’http: //code . google . com/p/pluralism/ 

2Receiver  annotations  could  be  moved  to  the  “receiver”  annotation  position  when  JSR-308  is  adopted.  JSR-308  proposes 
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@Param(name  =  "stmt",  releasedFrom("open")) 
public  interface  ResultSet  { 

@Full(guarantee  =  'open") 
@TrueIndicates("unread") 
@FalseIndicates("end") 
boolean  next(); 

@Full(guarantee  =  valid”,  ensures  =  "read") 
int  getlnt(int  column); 

@Pure(guarantee  =  "valid",  requires  =  ’read") 
boolean  wasNull(); 

@Full(ensures  =  closed") 

@Release("stmt") 
void  close(); 


Figure  6.1:  Simplified  ResultSet  specification  in  Plural  (using  the  typestates  shown  in  Figure  1.1). 

-  guarantee  declares  a  state  guarantee,  also  known  as  the  permission’s  root  node.  This  is  also 
the  default  attribute. 

-  requires  and  ensures  declare  required  and  ensured  states,  respectively. 

-  returned  can  be  set  to  false  to  declare  consumed  permissions;  otherwise,  the  permission  is 
borrowed  (Section  6.4.2). 

-  use  specifies  whether  the  permission  is  used  for  calling  other  methods  (default),  accessing  fields, 
or  both.  The  first  two  options  distinguish  virtual  and  frame  permissions  (see  Section  4.1.5);  the 
last  one  was  added  for  pragmatic  concerns  and  will  be  further  explained  in  Section  6.4.7. 

•  ©ResultUnique,  etc.,  declare  permissions  for  method  return  values  in  the  same  way  as  described 
above.  These  annotations  syntactically  distinguish  receiver  from  return  value  permissions. 

•  ©Truelndicates  and  ©Falselndicates  declare  a  method  to  be  a  dynamic  state  test  with  Boolean 
return  type,  where  the  return  value  true  {false,  respectively)  indicates  the  receiver  (when  placed  on  a 
method)  or  parameter  to  be  in  the  given  state  (Section  6.3.6). 

•  ©Perm  can  be  used  to  define  complex  method  pre-  and  post-conditions  with  the  textual  attributes 
requires  and  ensures.  Individual  permissions  are  expressed  with  the  syntax  we  use  in  the  formal 
system  (see  Chapter  4);  *.  &,  and  +  are  used  to  represent  linear  connectives;  and  concrete  predicates 
such  as  the  truth  of  a  Boolean  variable  can  also  be  asserted.  The  content  of  this  annotation  is  combined 
with  any  permission  annotations  (explained  above)  declared  for  the  same  method. 

•  ©Cases  is  used  to  declare  multiple  cases  for  a  single  method,  using  ©Perm  for  each  case. 


allowing  annotations  in  additional  syntactic  positions  including  for  method  receivers,  array  contents,  and  type  parameters  in  a 
future  version  of  Java.  Plural  could  benefit  from  all  of  these. 
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•  ©State  defines  a  state  invariant  using  the  same  textual  syntax  as  ©Perm.  This  annotation  is  aggregated 
with  ©ClassStates. 

•  ©Par am  declares  a  permission  parameter  (Section  6.4.5);  ©Capture  captures  and  ©Release  releases 
such  a  parameter  (see  Section  7.1.1  for  examples). 

•  ©Lend  annotates  a  parameter  to  “lend”  the  result  of  a  method,  temporarily  invalidating  that  parameter 
until  the  result  is  no  longer  used  (example  in  Figure  7.5). 

•  ©IsResult  can  be  used  to  declare  that  a  method  parameter  is  returned  as  the  method  result. 

Plural’s  annotations  for  license-technical  reasons  arc  homed  in  a  separate  open-source  project3  which 
contains  documentation  for  these  and  other  annotations. 


6.2  Background 

This  section  provides  background  on  the  Eclipse  IDE  and  the  Crystal  static  analysis  framework,  itself  a 
Eclipse  plug-in.  The  Plural  tool  relies  on  these. 

6.2.1  Eclipse 

Eclipse4  is  a  popular  open-source  integrated  development  environment  (IDE)  for  Java.5  It  provides  a  rich 
infrastructure  for  developing  plug-ins  that  augment  the  standard  Eclipse  features.  Two  capabilities  of  Eclipse 
are  essential  for  Plural  to  perform  its  task:  the  Java  AST  and  the  “problems  view”. 

Eclipse  provides  a  parser,  typechecker,  and  abstract  syntax  tree  (AST)  for  Java  source  files.  This  alle¬ 
viates  Crystal  and  Plural  from  having  to  parse,  typecheck,  and  represent  Java  code,  which  is  a  complicated 
undertaking  by  itself.  Since  it  is  well-tested,  the  AST  and  typing  information  provided  by  Eclipse  should 
be  (mostly)  reliable.  We  did  find  several  bugs  in  how  Eclipse  resolves  Java  5  annotations,  all  of  which  were 
fixed  with  Eclipse  release  3.4.1. 

Plural  uses  the  “problems  view”  to  communicate  possible  protocol  violations  to  the  developer.  This 
allows  Plural  to  conveniently  associate  problems  with  a  location  in  the  analyzed  Java  source  file  (usually  a 
method  or  constructor  call  site)  and  provide  a  textual  description  of  the  problem. 

6.2.2  Crystal 

Crystal6  is  a  static  analysis  framework  co-developed  by  the  author.  Crystal  is  itself  an  Eclipse  plug-in  that 
in  particular  takes  advantage  of  Eclipse’s  AST.  Crystal  provides  an  implementation  of  a  worklist  algorithm 
for  performing  dataflow  analyses  (Nielson  et  al.,  1999)  on  Java  source  files.  It  also  provides  an  abstraction 
of  Java  source  code  as  3-address  code. 

3-address  code  represents  nested  Java  expressions  as  sequences  of  atomic  instructions  that  assign  to 
temporary  variables  and  only  use  variables  as  operands.  Thanks  to  aggressive  desugaring,  the  many  different 

3http: //code . google . com/p/plaidannotations/ 

4http : / /www . eclipse . org/ 

’Eclipse  offers  support  for  editing  non-Java  artifacts,  but  these  capabilities  are  not  important  for  this  discussion. 

6http: //code . google . com/p/ crystalsaf / 


70 


CHAPTER  6.  PLURAL:  JAVA  TOOLING 


Java  expressions  can  be  represented  with  approximately  20  kinds  of  instructions.  This  reduction  simplifies 
writing  source  code  analyses  since  there  arc  fewer  cases  to  consider. 

Flow  analyses  in  Crystal  arc  defined  with  a  transfer  function.  A  transfer  function  takes  incoming  analysis 
information  and  a  “node”,  i.e.,  an  operation  in  the  control  flow  of  a  method,  and  computes  the  resulting  out¬ 
going  analysis  information.  Transfer  functions  are  analysis-specific,  and  Plural  implements  several  transfer 
functions  to  perform  its  task.  Crystal’s  worklist  algorithm  calls  a  given  transfer  function  to  transfer  over  the 
nodes  in  the  control  flow  graph  of  the  analyzed  method.  Transfer  functions  can  choose  to  transfer  directly 
over  Eclipse  Java  AST  nodes  or  on  3-address  code  instructions;  Plural  is  based  on  the  latter. 

Crystal’s  dataflow  analysis  infrastructure  provides  what  we  call  branch  sensitivity,  which  was  crucial 
for  supporting  dynamic  state  tests  in  Plural  (cf.  Section  6.3.6).  A  branch  is  a  point  in  the  control  flow  with 
multiple  successors  that  control  can  transfer  to  at  runtime.  Common  examples  for  branches  are  Boolean 
tests  and  operations  that  may  result  in  exceptions.  Crystal  “labels”  branches  with  the  conditions  under 
which  they  will  be  taken.  In  particular,  it  defines  labels  for  “true”  and  “false”  outcomes  of  Boolean  tests  and 
for  exceptions. 

When  Crystal’s  worklist  algorithm  invokes  a  transfer  function  to  transfer  over  a  branch  it  signals  the 
labels  on  the  branches  to  the  transfer  function.  The  transfer  function  then  has  to  opportunity  to  provide 
outgoing  analysis  information  separately  for  each  label.  For  example,  it  can  provide  separate  information 
for  the  “true”  and  “false”  branches  of  a  Boolean  test.  Crystal  will  propagate  information  associated  with  a 
label  only  on  branches  with  that  label,  which  will  ultimately  allow  analyzing  operations  following  Boolean 
tests  assuming  a  certain  test  outcome. 

Analysis  information  is  specific  to  a  transfer  function  and  is  therefore  defined  by  analysis  writers;  Crys¬ 
tal's  worklist  algorithm  is  parameterized  by  the  specific  analysis  information  being  tracked  by  a  given  trans¬ 
fer  function.  Two  operations  must  be  implemented  for  comparing  and  joining  analysis  information  at  control 
flow  merge  points.  Plural’s  implementation  of  these  operations  is  discussed  below. 

Branch  sensitivity  can  be  used  to  implement  path-sensitive  analyses:  at  control  flow  merge  points,  anal¬ 
ysis  writers  can  choose  to  keep  analysis  information  computed  for  the  different  incoming  paths  separate  by 
using  a  suitable  join  implementation.  However,  Plural  does  not  implement  such  a  join  and  instead  approxi¬ 
mates  analysis  information  from  different  paths  at  control  flow  merge  points. 


6.3  Flow  Analysis  for  Local  Permission  Inference 

Like  any  dataflow  analysis,  Plural  tracks  its  analysis  information  in  a  lattice.  Intuitively,  the  information 
Plural  tracks  arc  the  permissions  available  for  each  object.  In  Chapter  5  we  discussed  how  permissions 
can  be  inferred  automatically  for  a  program  using  constraints,  and  how  composite  contexts  allow  reasoning 
about  lineal-  logic  0  and  &  connectives.  Plural  tracks  constraints  as  part  of  its  flow  analysis  information  and 
includes  support  for  composite  contexts,  which  will  be  discussed  below.  Plural’s  transfer  functions  essen¬ 
tially  implement  the  permission  inference  rules  described  in  Chapter  5  and  will  therefore  not  be  discussed 
in  detail  here. 


6.3.1  Annotations  Make  Analysis  Modular 

Figure  6.2  shows  a  simple  client  method  that  retrieves  an  integer  value  from  the  first  column  in  the  first 
row  of  the  given  result  set.  Plural  can  be  used  to  check  that  this  code  respects  the  protocol  declared  for  the 
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public  static  Integer  getFirstInt(@Full(guarantee  =  "open")  ResultSet  rs)  { 
Integer  result  =  null; 
if(rs.next())  { 

result  =  rs.getlnt(l); 
if(rs.wasNull()) 
result  =  null; 
return  result; 


else  { 

return  rs.getlnt(l);  //ERROR:  rs  in  "end"  instead  of  "valid" 

} 


Figure  6.2:  Simple  ResultSet  client  with  error  in  else  branch  that  is  detected  by  Plural. 
ResultSet  interface  in  Figure  6.1. 

Our  goal  is  to  avoid  annotations  inside  method  bodies  completely:  based  on  the  declared  protocols, 
Plural  infers  how  permissions  flow  through  method  bodies.  Using  the  lattice  comparison  and  join  operations 
described  in  the  next  subsection,  Plural  can  automatically  infer  loop  invariants. 

However,  Plural  does  require  additional  annotations  on  method  parameters  that  have  a  declared  protocol, 
such  as  the  ResultSet  parameter  in  Figure  6.2.  We  annotate  client  code  with  the  same  annotations  that 
we  use  for  declaring  API  protocols.  While  protocol  annotations  on  the  API  itself  (e.g.,  Figure  6.1)  can 
conceivably  be  provided  by  the  API  designer  and  amortize  over  the  many  uses  of  that  API,  the  annotation 
shown  in  Figure  6.2  is  specific  to  this  client  program.  In  Section  7.4  we  discuss  the  overhead  of  providing 
these  additional  annotations  for  two  open-source  codebases. 

Annotations  make  the  analysis  modular:  Plural  checks  each  method  separately,  temporarily  trusting 
annotations  on  called  methods  and  checking  their  bodies  separately.  For  checking  a  given  method  or  con¬ 
structor,  Plural  assumes  the  permissions  required  by  the  method’s  annotations,  i.e.,  it  assumes  the  declared 
pre-condition.  At  each  call  site,  Plural  makes  sure  that  permissions  required  for  the  call  are  available,  splits 
them  off  (these  permissions  are  “consumed”  by  the  called  method  or  constructor),  and  merges  permissions 
ensured  by  the  called  method  or  constructor  back  into  the  current  context.  Notice  that  most  methods  “bor¬ 
row”  permissions  (cf.  Figure  6.1),  which  means  that  they  are  both  required  and  ensured.  At  method  exit 
points,  Plural  checks  that  permissions  ensured  by  its  annotations  are  available,  i.e.,  it  checks  the  declared 
post-condition. 

Thus,  permissions  are  handled  by  Plural  akin  to  conventional  Java  typing  information:  Permissions  are 
provided  with  annotations  on  method  parameters  and  then  tracked  automatically  through  the  method  body, 
like  conventional  types  for  method  parameters.  Unlike  with  Java  types,  local  variables  do  not  need  to  be 
annotated  with  permissions;  instead,  their  permissions  are  inferred  by  Plural.  Permission  annotations  can 
be  seen  as  augmenting  method  signatures.  They  do  not  affect  the  conventional  Java  execution  semantics; 
instead,  they  provide  a  static  guarantee  of  protocol  compliance  without  any  runtime  overhead. 

6.3.2  Tuple  Lattice 

At  the  heart  of  Plural’s  analysis  is  a  tuple  lattice  for  tracking  permissions  and  constraints  for  individual 
objects.  A  tuple  lattice  in  general  tracks  separate  analysis  information  for  each  object  and  is  compared  and 
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joined  pointwise.  The  information  tracked  for  each  object  is  a  pair:  the  set  of  permissions  currently  available 
for  the  object  (at  most  one  for  each  orthogonal  dimension)  together  with  the  constraints  collected  for  these 
permissions.7 

Such  a  tuple  conceptually  corresponds  to  an  atomic  context  as  discussed  in  Chapter  5.  However,  it  is  a 
simplification  because  atomic  contexts  can  in  general  contain  linear  logic  formulae,  whereas  Plural  tuples 
contain  atomic  permissions  only.  Formulae,  e.g.,  from  a  method  post-condition,  arc  broken  down  into  their 
constituent  permissions  before  merging  them  into  a  given  tuple.  This  has  no  consequences  on  precision  in 
the  case  of  multiplicative  conjunctions,  Pi  <g>  fA,  where  permissions  Pi  and  P2  can  be  separately  merged 
into  the  tuple.  But  this  strategy  can  result  is  a  loss  of  precision  when  inserting  ©  and  &  formulae.  However, 
we  never  explicitly  used  such  formulae  in  our  case  studies,  so  this  issue  has  had  no  practical  consequences.8 

Representing  individual  permissions.  Permissions  in  Plural  are  uniformly  represented  with  their  root 
node,  fraction  function,  state  information,  and  a  flag  distinguishing  read-only  permissions  (pure  and  im¬ 
mutable).  As  before,  the  fraction  function  is  a  mapping  from  nodes  in  the  state  hierarchy  to  fractions 
together  with  a  “below”  fraction.  Permissions  declared  in  annotations  will  contain  literal  fractions  such  as 
1  and  0,  while  permissions  generated  by  permission  inference  rules  will  usually  contain  fraction  variables 
only. 


6.3.3  Comparing  and  Joining  Permission  Tuples 

When  comparing  sets  of  permissions  for  the  same  object  (as  mentioned,  tuples  arc  compared  and  joined 
pointwise).  Plural  pairs  up  permissions  in  the  two  sets  with  the  same  root  node  (or  related  root  nodes).  It 
then  compares  these  permissions  based  on  several  criteria  including  the  following: 

1.  Additional  permissions  make  that  permission  set  more  precise.  If  both  sets  contain  permissions  for 
nodes  not  contained  in  the  other  set  then  the  sets  are  incomparable. 

2.  When  comparing  permissions  with  the  same  root  it  compares  fractions  for  the  same  node  pairwise 
and  treats 

•  pairwise  equal  fractions  in  the  fraction  function  as  equal, 

•  logic  variable  (Z)  or  literal  fractions  (such  as  1)  in  place  of  unknown  fractions  (z)  from  existential 
quantifiers  in  the  other  permission  as  more  precise9,  and 

•  all  other  fractions  as  incomparable. 

3.  if  there  are  permissions  for  subnodes  of  a  node  in  one  set  for  which  the  other  set  contains  a  permission 
for  that  node.  Plural  attempts  to  align  these  permissions  by  moving  roots  up  or  down  and  then  use  the 

7In  Chapter  5  we  used  one  set  of  constraints,  but  Plural  keeps  constraints  for  different  objects  separate  to  simplify  satisfiability 
checking  and  error  reporting. 

sDynamic  state  tests  and  method  cases  (Sections  6.3.6  and  6.4.3)  are  treated  specially  and  do  not  suffer  from  these  imprecisions. 

9Care  must  be  taken  not  to  treat  unknown  fractions  front  the  checked  method’s  pre-condition  as  less  precise  than  other  fractions, 
as  this  would  compromise  our  treatment  of  borrowed  permissions  (Section  6.4.2).  The  difference  is  that  unknowns  from  pre¬ 
conditions  result  from  unpacking  universal  quantifiers,  while  the  unknowns  introduced  in  joining  permissions  result  from  existential 
quantifiers,  which  can  be  alpha-converted. 
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above  rules  to  compare  the  new  sets;  if  root  movements  arc  not  possible  then  the  permission  sets  arc 
considered  incomparable. 

The  same  rules  guide  the  process  of  joining  permission  sets:  excess  permissions  arc  dropped,  incompa¬ 
rable  pairs  of  permissions  arc  approximated  (described  below),  and  permissions  with  misaligned  nodes  arc 
either  aligned  (through  root  movement)  or  dropped  (if  alignment  is  not  possible). 

A  pair  of  incomparable  permissions  is  approximated  by  generating  a  new  permission  with  fresh  unknown 
fractions  (in  contrast  to  the  fresh  variable  fractions  introduced  by  the  permission  inference).  Because  these 
fresh  unknown  fractions  have  an  unknown  value  they  arc  deemed  less  precise  than  variable  and  literal 
fractions,  as  mentioned  in  the  description  of  Plural’s  comparison  rules. 

Notice  that  we  do  not  attempt  to  determine  whether  a  fraction  is  smaller  than  another  given  a  set  of 
constraints,  although  this  would  allow  comparison  of  permissions  incomparable  in  the  above  rules  (larger 
fractions  arc  more  precise).  We  have  not  found  this  to  be  a  problem  in  practice,  but  it  appeal's  that  such  a 
refinement  of  our  lattice  comparison  algorithm  would  be  possible. 

Our  strategy  of  introducing  unknown  fractions  to  join  incomparable  permissions  marks  a  difference  to 
previous  work  on  inferring  fractions  (Terauchi  and  Aiken,  2008;  Terauchi,  2008).  Previously,  control  flow 
merges  introduced  new  fraction  variables  that  were  forced  to  be  smaller  than  the  incoming  fractions.  Func¬ 
tion  signatures  were  also  inferred  with  fraction  variables.  Constraints  could  then  be  collected  by  scanning 
the  entire  program  once,  even  in  the  presence  of  conditionals  and  loops,  allowing  the  process  of  constraint 
collection  to  be  asymptotically  linear  in  the  size  of  the  program  and  avoiding  the  complications  of  deciding 
constraints  with  unknown  variables  (see  Section  5.2.2). 

Conversely,  we  collect  constraints  with  a  flow  analysis,  which  is  polynomial  in  the  worst  case  (Nielson 
et  al.,  1999,  Chapter  6).  We  could  nonetheless  use  the  previous  approach  for  joining  permissions  for  control 
flow  merges  resulting  from  if  statements,  but  it  would  not  terminate  for  loops.  An  advantage  of  our  approach 
is  that  permissions  can  be  consumed  in  loops,  which  was  not  previously  possible  (see  Terauchi,  2008, 
Section  5).  While  most  methods  do  not  consume  permissions,  some  do  in  the  APIs  we  investigated  (see 
Chapter  7),  usually  those  methods  related  to  creating  new  objects  based  on  existing  ones.  We  are  also  able 
to  use  universal  quantification  in  function  signatures  (currently  provided  with  explicit  annotations),  which 
may  be  more  flexible  than  inferring  concrete  fractions  for  signatures,  as  was  done  previously. 

Tuple  lattice  has  finite  height.  As  with  any  dataflow  analysis,  the  question  arises  whether  ours  is  guaran¬ 
teed  to  terminate,  which,  because  of  the  iterative  nature  of  the  algorithm,  can  be  reduced  to  showing  that  the 
lattice  in  use  has  finite  height.  We  informally  argue  that  this  is  the  case. 

•  Tuple  lattices  have  finite  height  (for  a  finite  set  of  objects)  if  the  information  tracked  for  an  individual 
object  forms  a  lattice  of  finite  height.  In  our  case,  this  information  is  the  permission  set  tracked  for  an 
object. 

•  Our  permission  sets  are  conceptually  akin  to  a  set  intersection  lattice,  where  the  empty  set  represents 
the  least  amount  of  information  (notice  that  we  drop  permissions  present  in  one  but  not  the  other  set 
when  joining  sets).  The  size  of  our  sets  is  finite  because  there  can  be  only  at  most  one  permission 
rooted  in  each  node  of  the  tracked  object’s  state  space  in  it,  and  that  state  space,  being  user-defined,  is 
finite.  (Recall  that  permission  inference  merges  permissions  with  the  same  root  node,  see  Chapter  5.) 
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•  When  joining  individual  permissions  we  either  (a)  preserve  the  fractions  present  in  these  permissions 
(if  they  arc  equal)  or  (b)  approximate  them  with  an  unknown  variable.  Already  approximated  fractions 
will  not  be  approximated  again  because  the  unknowns  that  were  introduced  arc  less  precise  than  any 
other  fraction  (including  other  unknowns),  terminating  the  process  of  approximation  of  individual 
permissions  after  one  iteration. 

Future  work:  interprocedural  analysis.  We  believe  that  the  approach  for  comparing  and  joining  per¬ 
mission  tuples  described  above  could  be  used  in  an  interprocedural  analysis  as  well.  In  particular,  methods 
being  called  from  multiple  call  sites  will  cause  the  tuples  available  at  the  different  call  sites  to  be  joined, 
possibly  introducing  unknown  variables,  which  directly  represent  the  universally  quantified  fractions  we 
typically  find  in  developer-defined  method  pre-conditions.  Such  an  approach  appears  to  have  similarities  to 
let-polymorphism  in  Hindley-Milner  type  inference  (Pierce,  2002). 

This  would  allow  automatically  analyzing  larger  program  modules,  such  as  a  set  of  related  classes  or  an 
entire  program.  However,  constraint  sets  would  possibly  grow  to  very  large  sizes,  which  has  been  a  problem 
in  previous  work  when  analyzing  a  whole  program  at  once  (Terauchi,  2008). 

6.3.4  Composing  Tuples 

The  tuple  lattice  discussed  above  induces  a  lattice  of  composite  tuples  that  correspond  to  the  composite 
choice  and  all  contexts  from  Section  5.1.  Formally,  we  define  the  elements  A  of  this  lattice  as  follows: 

A,B,C  ::=  T  atomic  tuple 

A  Sz  B  choice  element 
A®  B  all  element 
T  top 

_L  bottom 

Comparing  the  elements  of  this  lattice  is  straightforward,  given  the  comparison  Ti  C  T2  for  individual 
tuples  we  described  in  Section  6.3.3. 

ACC  BCC  AC  B  ACC 

ASzBCC  ASzBCC  AC  B IV  C  ACT 

ACC  BCC  AC  B  ACC 

A®  B  C  C  A  C  B  ®  C  AC  B  ®  C  i  C  A 

These  correspond  exactly  to  the  subtyping  rules  for  union  and  intersection  types,  where  A  Sz  B  corre¬ 
sponds  to  intersections  and  A®  B  corresponds  to  unions  (Dunfield  and  Pfenning,  2004). 

This  lattice  allows  reasoning  about  linear  logic  predicates  that  include  internal  (&)  and  external  choice 
(0)  operators.  Unfortunately,  however,  the  lattice  has  infinite  height.  We  can  finitize  it  by  limiting  the 
nesting  depth  of  tuples.  For  example,  we  can  restrict  ourselves  to  only  allow  atomic  tuples  in  &>elements 
and  only  ^-elements  in  0-elements,  similar  to  a  disjunctive  normal  form.  Such  a  restriction  is  not  complete 
with  respect  to  the  infinite  lattice  because  of  our  limitation  to  a  fragment  of  linear  logic.  In  particular, 
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the  problem  is  that  in  the  constructive  fragment  we  arc  using  (which  does  not  include  negation),  certain  De 
Morgan  rules  only  work  in  one  direction  but  not  the  other.  For  example  (.4  &  B)  %C  h  (A®C)  8z  (B®C), 
but  (A  <g>  C)  &  (B  <g>  C)  \f  (A  &  B)  <g>  C. 

The  implementation  of  Plural  is  currently  restricted  to  only  use  ^-elements.  This  simplification  was 
possible  because  we  only  used  conjunctions  in  our  case  studies,  never  internal  or  external  choices.  Fur¬ 
thermore,  we  suspect  that  the  added  benefit  of  0-elements  would  be  small  because  conventional  joining  of 
lattice  elements  achieves  something  very  similar  to  0-elements.  Joining  approximates  two  elements  into  one 
that  contains  as  much  information  from  the  two  original  elements  as  possible.  The  resulting  lattice  element 
must  then  be  able  to  satisfy  the  subsequent  predicates.  With  0-elements,  Plural  still  has  to  make  sure  that 
both  alternatives  satisfy  all  subsequent  predicates,  ^-elements,  on  the  other  hand,  allow  delaying  choices, 
which  is  in  particular  useful  in  the  context  of  API  implementation  checking  and  for  handling  method  cases. 

6.3.5  Local  Must  Alias  Analysis 

Plural  uses  a  simple  local  alias  analysis  to  identify  local  variables  that  are  known  to  point  to  the  same  object. 
The  local  alias  analysis  tracks  a  set  of  “locations”  for  each  local  variable.  These  locations  are  effectively 
Plural’s  static  approximation  of  the  different  objects  that  a  local  variable  may  point  to  at  run  time.  The  tuple 
lattice  described  above  therefore  associates  permission  sets  and  constraints  with  these  locations. 

Plural  generates  fresh  locations  for  references  returned  from  method  calls.  These  references  may  at  run¬ 
time  point  to  the  same  object  as  another  local  variable.  In  other  words,  local  variables  may  alias  even  though 
our  local  alias  analysis  does  not  indicate  this  possibility.  The  use  of  permissions  prevents  problems  in  this 
case  because  Plural  will  associate  different  sets  of  permissions  with  the  two  locations  which  correspond  to 
the  same  runtime  object. 

The  puipose  of  Plural’s  local  alias  analysis  is  therefore  not  to  soundly  approximate  all  possible  runtime 
aliasing,  but  to  identify  local  variables  that  must  alias  and  can  therefore  access  the  same  permissions.  The 
local  alias  analysis  in  particular  prevents  us  from  having  to  split  permissions  when  a  local  is  assigned  to  an¬ 
other  local  with  a  copy  instruction,  such  as  x  =  y,  which  avoids  developer-provided  annotations  in  method 
bodies. 


6.3.6  Dynamic  State  Tests 

APIs  often  include  methods  whose  return  value  indicates  the  current  state  of  an  object,  which  we  call  dy¬ 
namic  state  tests.  For  example,  next  in  Figure  6. 1  is  specified  to  return  true  if  the  cursor  was  advanced  to  a 
valid  row  and  false  otherwise.  Sensitivity  to  such  tests  is  paid  of  the  type  system  underlying  Plural  (Chapter 
4);  this  section  discusses  how  Plural  automatically  reasons  about  dynamic  state  tests  using  Crystal’s  support 
for  branch  sensitivity. 

In  order  to  take  such  tests  into  account.  Plural  performs  a  branch- sensitive  flow  analysis:  if  the  code 
tests  the  state  of  an  object,  for  instance  with  an  if  statement,  then  the  analysis  updates  the  state  of  the  object 
being  tested  according  to  the  test’s  result.  For  example,  Plural  updates  the  result  set’s  state  to  unread  at 
the  beginning  of  the  outer  if  branch  in  Figure  6.2.  Likewise,  Plural  updates  the  result  set’s  state  to  end  in 
the  else  branch  and,  consequently,  signals  an  error  on  the  call  to  getlnt.  Section  6.4.6  explains  how  these 
lattice  manipulations  are  implemented  internally. 

Notice  that  this  approach  does  not  make  Plural  path-sensitive:  analysis  information  is  still  joined  at 
control-flow  merge  points.  Thus,  at  the  end  of  Figure  6.2,  Plural  no  longer  remembers  that  there  was  a  path 
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through  the  method  on  which  the  result  set  was  valid.  We  believe  that  Plural  could  be  extended  to  retain  this 
information,  but  then  we  would  have  to  deal  with  the  usual  complications  of  path  sensitivity,  i.e.,  large  or 
infinite  numbers  of  paths  even  through  small  methods. 

When  checking  the  implementation  of  a  state  test  method.  Plural  checks  at  every  method  exit  that, 
assuming  true  (or  false)  is  returned,  the  receiver  is  in  the  state  indicated  by  true  (resp.  false).  This 
approach  can  be  extended  to  other  return  types,  although  reasoning  about  predicates  such  as  integer  ranges 
may  require  using  a  theorem  prover  (Bierhoff  and  Aldrich,  2008;  Leino  and  Muller,  2009). 

6.3.7  API  Implementation  Checking 

Our  approach  not  only  allows  checking  whether  a  client  of  an  API  follows  the  protocol  required  by  that 
API,  it  can  also  check  that  code  implementing  an  API  is  safe  when  used  with  its  declared  protocol.  The  key 
abstraction  for  this  arc  state  invariants,  which  we  adapted  from  Fugue  (DeLine  and  Fahndrich,  2004b,  see 
also  Chapter  3.2).  A  state  invariant  associates  a  typestate  of  a  class  with  a  predicate  over  the  fields  of  that 
class.  In  our  approach,  this  predicate  usually  consists  of  access  permissions  for  fields.  An  example  can  be 
found  in  Figure  7.7. 

Whenever  a  receiver  field  is  used.  Plural  unpacks  a  permission  to  the  surrounding  object  to  gain  access 
to  its  state  invariants  (Bierhoff  and  Aldrich,  2007b).  Essentially,  unpacking  means  replacing  the  receiver 
permission  with  permissions  for  the  receiver’s  fields  as  specified  in  state  invariants.  Before  method  calls, 
and  before  the  analyzed  method  returns.  Plural  packs  the  receiver,  possibly  to  a  different  state,  by  splitting  off 
that  state’s  invariant  from  the  available  field  permissions  (Bierhoff  and  Aldrich,  2007b).  Plural  sometimes 
has  to  “guess”  what  state  to  pack  to  by  frying  all  possibilities,  which  our  “choice”  contexts  allow  us  to  do. 

6.3.8  Error  Reporting 

Plural  reports  possible  protocol  violations  at  the  point  in  the  program  where  errors  arc  detected.  In  particular, 
if  a  method  pre-condition  cannot  be  satisfied  at  a  call  site  then  Plural  will  issue  a  warning  at  that  call  site, 
explaining  the  nature  of  the  violation.  Plural  finds  violations  by  querying  the  results  of  the  dataflow  analysis 
and  checking  at  each  method  call  site  whether  the  pre-condition  is  satisfiable.  Note  that  the  actual  error  may 
precede  the  point  in  the  method  where  a  protocol  violation  occurs,  for  instance  because  an  earlier  method 
call  may  set  up  a  situation  where  a  later  pre-condition  becomes  unsatisfiable. 

6.4  Extensions 

This  section  describes  features  implemented  in  Plural  that  are  not  captured  in  the  theory  underlying  the  tool 
but  which  we  found  useful  in  practice  (see  Chapter  7). 

6.4.1  Immutable  Permissions 

While  not  paid  of  the  theory  for  permission  reasoning  presented  in  the  preceding  sections,  we  chose  to  in¬ 
clude  immutable  permissions  in  Plural  because  of  their  apparent  relevance  notably  for  specifying  collections 
and  iterators  (Section  3.1).  We  track  immutable  exactly  like  share  permissions;  the  only  difference  is  that  we 
will  not  allow  assignments  to  fields  when  unpacking  an  immutable  permission.  Therefore,  the  simple  “read¬ 
only”  flag  mentioned  in  Section  6.3.2  is  sufficient  for  distinguishing  immutable  from  share  permissions  in 
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the  lattice.  However,  we  must  avoid  immutable  and  share  permissions  from  co-existing;  in  particular,  we 
must  never  split  off  an  immutable  from  a  share  permission  or  vice  versa.  Instead,  we  must  re-assemble  a 
full  (or  stronger)  permission  before  switching  from  share  to  immutable  or  the  other  way  around. 

In  terms  of  constraints  this  means  that  splitting  a  share  from  a  read-only  or  a  immutable  from  a  mutable 
permission  p,  we  generate  an  additional  constraint  that  forces  p  to  be  full,  i.e.,  we  force  its  “below”  fraction 
to  be  1.  This  is  because  a  full  permission  is  required  to  transition  between  objects  immutable  and  objects 
being  share. 

6.4.2  Borrowing 

Over  and  over  again  we  encountered  methods  that  borrow  permissions,  meaning  they  return  the  same  per¬ 
mission  that  was  passed  in  when  they  were  invoked,  although  possibly  with  new  state  information.10  Tech¬ 
nically,  our  type  system  cannot  express  borrowing  because  it  only  quantifies  over  fractions  inside  a  method 
pre-  or  post-condition,  but  over  not  both.  Borrowed  permissions  are  essentially  permissions  whose  fractions 
are  quantified  over  across  both  the  method  pre-  and  post-condition.*  1 1 
Support  for  borrowing  has  two  important  implications: 

•  Precision.  When  a  method  call  requires  to  move  the  root  of  a  permission  available  at  the  call  site 
down,  and  this  permission  is  borrowed,  then  it  is  always  safe  to  move  the  root  back  to  where  it  was 
before  the  call.  Without  support  for  borrow  ing,  the  following  code  would  flag  an  error,  while  Plural 
(correctly)  does  not  flag  an  error  at  the  call  to  remove. 

public  static  <T>  T  removeNext( @Full(requires  =  "available")  Iterator<T>  it)  { 

T  result  =  it.next(); 

it.remove() ;// annotated  with  @Full( guarantee  =  "current") 

return  result; 

} 

The  reason  is  that  the  given  full  permission  is  strong  enough  to  move  the  root  from  alive  to  current, 
but  not  back  up  (moving  a  root  up  requires  a  unique  permission).  With  borrow  ing,  we  know  that  we 
can  move  the  root  back  to  where  it  was. 

•  Performance.  Support  for  borrowing  also  seems  to  improve  performance:  it  cut  the  time  Plural  took 
to  run  over  its  regression  test  suite  in  half  (from  70  to  about  35  seconds  on  the  author’s  machine).  This 
is  because  we  do  not  have  to  keep  constraints  introduced  for  borrowed  permissions  beyond  the  current 
call  site  and  can  instead  revert  to  the  permission  that  was  previously  in  the  lattice,  although  possibly 
with  changed  state  information.  Thus,  borrowed  permissions  prevent  constraints  from  accumulating 
over  consecutive  call  sites,  which  simplifies  determining  whether  constraints  are  satisfiable  in  the  end. 
(This  optimization  is  only  sound  if  constraint  violations  are  detected  immediately,  which  Plural  does.) 

6.4.3  Method  Cases 

The  idea  of  method  cases  goes  back  to  behavioral  specification  methods,  e.g.,  in  the  JML  (Leavens  et  al., 
1999).  Method  cases  amount  to  specifying  the  same  method  with  multiple  pre-/post-condition  pairs,  al¬ 
lowing  methods  to  behave  differently  in  different  situations.  We  early  on  recognized  their  relevance  for 

10The  other  common  case  is  that  permissions  are  consumed,  i.e.  not  returned  to  the  caller  of  a  method. 

11  Such  quantifiers  could  be  added  to  the  type  system  with  additional  notational  overhead  in  the  rules  for  checking  method  calls. 
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specifying  API  protocols  (Bierhoff  and  Aldrich,  2005;  Bierhoff,  2006),  but  we  arc  not  aware  of  any  other 
protocol  checking  approaches  that  support  method  cases. 

In  terms  of  linear  logic,  method  cases  can  be  thought  of  as  a  choice  between  implications.  For  example, 
(Pi  — o  P2)  &  (P3  — o  P4)  encodes  a  method  with  two  cases  Pi  — o  P2  and  P3  — o  P4. 

In  order  to  support  method  cases,  Plural  tracks  the  possible  permissions  after  a  call  to  a  method  with 
cases  using  “choice”  lattice  elements.  Checking  the  implementation  of  a  method  with  multiple  cases 
amounts  to  checking  the  method  separately  for  each  case. 

6.4.4  Marker  States 

Plural  can  treat  states  special  that  are  fixed  throughout  the  object  lifetime.  We  call  these  marker  states,  which 
are  reminiscent  of  (flow-insensitive)  type  qualifiers  (Foster  et  ah,  2002)  and  type  refinements  (Dunfield  and 
Pfenning,  2004).  For  example,  result  sets  can  be  marked  as  updatable  or  readonly  (see  Section  7.1.1),  and 
they  cannot  switch  from  one  to  the  other  once  created.  Knowledge  about  an  object  being  in  a  marker  state, 
once  gained,  cannot  be  lost,  which  can  simplify  checking  API  clients.  Otherwise,  marker  states  are  techni¬ 
cally  no  different  from  regular  states.  But  they  may  also  be  informative  to  a  human  reading  a  specification: 
marker  states  indicate  object  properties  that  are  fixed  at  construction  time,  thereby  directly  refining  conven¬ 
tional  Java  types  with  additional,  flow-insensitive  information  that  does  not  change  throughout  the  object’s 
lifetime. 

Notably,  marker  states  can  capture  the  well-known  distinction  between  “modifiable”  and  “unmodifi- 
able”  collections  as  they  are  defined  in  the  Java  Collections  Framework.  We  can  also  use  marker  states  for 
distinguishing  “readonly”  and  “modifying”  iterators  over  mutable  collections  (Section  7.1.2). 

6.4.5  Dependent  Objects 

Another  feature  of  many  APIs  is  that  objects  can  become  invalid  if  other,  related  objects  are  manipulated  in 
certain  ways.  For  example,  SQL  query  results  become  invalid  when  the  originating  database  connection  is 
closed.  (A  similar  problem,  called  concurrent  modification,  exists  with  iterators  (Bierhoff,  2006).)  There  are 
no  automated  modular  protocol  checkers  that  we  know  of  that  can  handle  these  protocols,  although  recent 
whole-program  protocol  checking  approaches  can  (Bodden  et  ah,  2008;  Naeem  and  Lhotak,  2008). 

Permission  Parameters.  Our  solution  is  to  “capture”  a  permission  in  the  dependent  object  (the  result  set 
in  the  example)  which  prevents  the  problematic  operation  (closing  the  connection  in  the  example)  from 
happening.  The  dependent  object  has  to  be  invalidated  before  “releasing”  the  captured  permission  and  re¬ 
enabling  the  previously  forbidden  operation.  Captured  permissions  are  declared  in  Plural  with  a  ©Param 
annotation,  and  ©Release  explicitly  releases  permissions,  as  in  close  (Figure  6.1). 

Garbage  Collection.  Others  have  modeled  dependent  objects  with  linear  implications  (Boyland  and  Retert, 
2005;  Krishnaswami,  2006;  Haack  and  Hurlin,  2008)  but  it  is  unclear  how  well  those  approaches  can  be  au¬ 
tomated.  Our  solution  is  to  use  a  live  variable  analysis  to  detect  dead  objects,  i.e.,  dead  references  to  objects 
with  unique  permissions,  and  release  any  captured  permissions  from  these  dead  objects.12 

12We  could  check  that  these  objects  are  deleted  (in  C  or  C++)  or  mark  them  as  available  for  garbage  collection  (in  Java  or  C#), 
but  we  are  not  exploring  this  optimization  possibility  here. 
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6.4.6  Concrete  Predicates 

With  concrete  predicates  we  mean  information  about  values  such  as  primitive  Booleans  as  well  as  the  null¬ 
ness  of  references.  Tracking  the  former  is  necessary  for  properly  handling  dynamic  state  tests,  while  the 
latter  is  useful  in  implementation  predicates  as  well  as  for  avoiding  null  dereference  errors.  At  the  end  of 
this  section  we  discuss  how  integers  could  be  tracked. 

Every  test  in  a  program,  for  example  in  an  if  statement  or  loop  header,  ascertains  the  truth  or  falsehood 
of  some  (temporary)  variable  at  run  time.  This  information  often  implies  or  indicates  some  other  fact. 
For  example,  truth  of  the  JDBC  ResultSet’s  next  method’s  return  value  indicates  that  the  method  call’s 
receiver  is  in  the  valid  state  (see  Figure  6.1). 

Plural’s  lattice  pairs  the  tuples  described  above  with  a  “dynamic  state  logic”  that  maintains  (a)  knowl¬ 
edge  about  concrete  predicates  (truth,  falsehood,  and  nullness)  and  (b)  a  list  of  known  implications.  The 
latter  mirror  dynamic  state  test  annotations  such  as  (sTruelndicates  with  implications  from  the  indicat¬ 
ing  Boolean  value  to  the  indicated  state.  Plural  leverages  Crystal’s  support  for  branch  sensitivity  to  pass 
truth  and  falsehood  of  the  tested  variable  down  the  respective  branches.  It  eagerly  eliminates  applicable 
implications  when  knowledge  about  the  value  of  a  Boolean  variable  becomes  available. 

Plural  handles  the  fact  that  a  reference  is  “null”  or  “non-null”  similar  to  a  Boolean  fact  about  a  variable. 
If  null-ness  is  tested  at  run  time,  Plural  produces  an  implication  from  a  Boolean  to  a  null-ness  fact  as 
described  above. 


Future  Work:  Integers.  Facts  about  integer  variables,  such  as  an  integer  being  positive,  could  conceiv¬ 
ably  be  tracked  by  Plural  as  well.  Unlike  with  Boolean  and  null-ness  facts,  however,  integers  arc  notoriously 
difficult  to  reason  about,  and  facts  about  them  arc  hard  to  track  precisely  and  efficiently.  In  the  spirit  of  Plu¬ 
ral,  one  could  use  sufficiently  precise  and  efficient  lattices  to  track  integer  ranges,  as  in  recent  work  by 
Ferrara  et  al.  (2008).  Alternatively,  one  could  employ  a  theorem  prover,  as  is  common  in  program  verifica¬ 
tion  (Flanagan  et  ah,  2002),  which  would  increase  reasoning  power  but  also  overhead  and  might  decrease 
predictability. 


Discussion.  Typestates  arc  a  promising  framework  for  tracking  concrete  predicates  more  precisely.  What 
we  mean  is  that  the  tracking  of  concrete  predicates  discussed  above  is  a  well-studied  area,  often  even  com¬ 
monly  tracked  with  dataflow  analyses  based  on  simple  lattices.  But  we  found  that  concrete  predicates  often 
depend  on  an  object’s  typestate:  for  instance,  a  field  may  only  be  non-null  in  a  certain  state  and  otherwise  be 
null.  We  therefore  believe  that  wrapping  a  typestate  framework,  such  as  Plural’s,  around  these  well-known 
techniques  for  tracking  concrete  predicates  would  allow  tracking  them  in  the  appropriate  context  (e.g.,  as¬ 
suming  a  particular  typestate)  and  yield  higher  precision.  In  particular,  considerable  research  effort  has  gone 
into  reasoning  about  object  initialization  (Fahndrich  and  Xia,  2007;  Qi  and  Myers,  2009),  which  we  believe 
can  be  largely  encoded  with  typestates. 

At  the  same  time,  concrete  predicates  arc  crucial  for  reasoning  about  dynamic  state  tests,  thereby  helping 
our  typestate  analysis.  While  Plural  only  supports  state  test  methods  of  Boolean  return  type,  we  have  seen 
integers  and  null-ness  as  well  as  regular  or  exceptional  control  flow  indicate  states. 
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6.4.7  Permissions  Allowing  Dispatch  And  Field  Access 

In  Section  4.1.5  we  saw  how  a  distinction  into  frame  and  virtual  permissions  allows  using  permissions 
smoothly  with  inheritance.  Roughly,  frame  permissions  allow  field  access  (only),  and  virtual  permissions 
allow  calling  other  methods  using  dynamic  dispatch  (only).  For  a  caller,  this  distinction  is  irrelevant:  virtual 
permissions  have  to  be  provided  by  callers  in  either  case,  since  virtual  permissions  arc  coerced  into  frame 
permissions  as  needed. 

In  our  case  studies  we  encountered  code  which  accessed  fields  and  called  other  methods  in  the  body  of 
the  same  method.  In  particular,  the  restriction  to  one  or  the  other  accounted  for  42  out  of  77  false  positives 
in  the  parts  of  PMD  we  considered  for  one  of  our  case  studies  (see  Section  7.3.2). 

Sometimes,  we  can  use  separate  permissions  to  specify  these  methods,  for  instance  a  pure  frame  permis¬ 
sion  for  reading  fields  and  a  full  virtual  permission  for  calling  another  method  that  required  it.  But  at  other 
times,  a  full  frame  permission  will  be  needed  to  modify  fields  or  the  content  of  fields,  and  a  full  virtual  per¬ 
mission  is  required  by  called  methods.  Such  a  method  pre-condition,  however,  can  never  be  satisfied  because 
callers  would  have  to  provide  two  full  permissions  for  the  same  object,  which  is  by  design  impossible. 

We  remedied  this  problem  by  introducing  permission  annotations  allowing  both  field  access  and  dy¬ 
namic  dispatch.  Plural  checks  implementations  of  methods  annotated  with  these  permissions  twice.  These 
two  checks  correspond  to  the  two  situations  under  which  a  given  method  could  be  invoked:  the  receiver, 
this,  is  at  runtime  either  an  instance  of  the  class  that  the  method  is  declared  in  or  an  instance  of  a  subclass. 
Other  situations  cannot  occur.  Plural  now  checks  both  situations  separately,  both  times  encoding  these 
fields-and-dispatch  permissions  with  regular  frame  and/or  virtual  permissions. 

•  When  the  receiver  is  an  instance  of  the  class  that  the  method  is  declared  in  then  the  method  is  in  fact 
accessing  the  virtual  frame  of  the  receiver,  i.e.,  the  frame  corresponding  to  the  runtime  type  of  the 
receiver.  In  this  case,  frame  and  virtual  permissions  in  fact  grant  access  to  the  same  frame.  Plural 
checks  this  case  by  using  a  single  (frame)  permission  of  the  declared  kind  for  accessing  fields  and 
dynamic  dispatch. 

•  When  the  receiver  is  not  an  instance  of  the  class  that  the  method  is  declared  in  then  frame  and  virtual 
permissions  in  fact  grant  access  to  different  frames.  Plural  checks  this  case  by  using  both  a  frame  and 
a  virtual  permission  of  the  declared  kind,  the  former  allowing  field  accesses  and  the  latter  allowing 
dynamic  dispatch. 

Calling  a  thus  annotated  method  using  dynamic  dispatch  will  require  a  virtual  permission  of  the  declared 
kind  at  the  call  site.  Just  as  with  frame  permissions,  we  assume  that  methods  requiring  permissions  for  both 
field  access  and  dynamic  dispatch  are  overridden.  Therefore,  a  method  can  only  be  invoked  in  the  second 
situation  discussed  above  by  a  subclass  through  a  super- call,  for  which  the  subclass  will  be  required  to 
provide  separate  super- frame  and  virtual  permissions  for  the  receiver  as  before.  Implementing  this  extension 
removed  31  out  of  77  false  positives  in  checking  parts  of  PMD  (Section  7.3.2). 

This  extension  allows  checking  methods  as  if  the  surrounding  class  was  final.  But  we  also  allow  sub¬ 
classing.  Figure  6.3  illustrates  how  a  method  requiring  a  full  permission  can  be  overridden  and  called  in  a 
subclass.  (If  the  overridden  method  is  never  called  or  only  requires  “duplicable”  permissions  such  as  share 
then  overriding  should  be  straightforward.) 
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class  C  { 

@Full(use  =  Use.DISP_FIELDS) 
void  m()  {  ...  } 

} 


@ClassStates(@State(name  =  "alive",  inv  =  "quiet  ==  true  =>  full(super)")) 
class  D  extends  C  { 
private  boolean  quiet  =  true; 

void  m()  {  if(quiet)  {  quiet  =  false;  super.m();  quiet  =  true;  }  } 

} 


Figure  6.3:  Overriding  a  method  requiring  a  full  “dispatch-and-fields”  permission 


6.5  Dealing  with  Java 

This  section  details  how  Plural  handles  some  of  the  Java  features  that  are  not  covered  explicitly  by  the 
underlying  theory  (cf.  Chapter  4),  in  particular,  constructors  and  arrays.  We  also  discuss  some  aspects  of 
Java  that  are  not  currently  handled  sufficiently.  All  of  the  features  discussed  in  this  section  are  common  in 
practical  programming  languages;  therefore,  the  discussion  applies  to  languages  other  than  Java  as  well. 


6.5.1  Constructors 

We  leverage  API  implementation  checking  (Section  6.3.7)  to  reason  about  constructors  by  injecting  an 
unpacked  unique  permission  for  the  receiver  frame  into  the  initial  lattice  element,  in  addition  to  any  explicitly 
required  permissions  for  constructor  arguments.  Permissions  needed  for  virtual  method  invocations  from 
inside  a  constructor  have  to  be  declared  as  usual;  these  permissions  are  checked  when  subclass  constructors 
invoke  the  superclass  constructor.  This  means  that  constructors  have  to  admit  to  any  virtual  method  calls 
they  perform,  which  appears  desirable  considering  the  amount  of  grief  such  calls  can  cause  to  subclass 
developers. 


6.5.2  Private  Methods 

Private  methods  can  by  definition  not  be  overridden.  This  means  that  the  frame  they  work  on  is  stati¬ 
cally  known  to  be  the  class  they  are  defined  in.  Therefore,  Plural  handles  calls  to  private  methods  akin  to 
super- calls  (cf.  Chapter  4):  frame  permissions  required  by  a  private  method  must  be  satisfied  with  frame 
permissions  available  at  the  call  site.  This  simplifies  specifying  the  methods  calling  private  methods  because 
they  typically  do  not  need  to  require  both  a  frame  (for  their  own  field  accesses)  and  a  virtual  permission  (for 
calling  the  private  method)  but  just  a  frame  permission. 

6.5.3  Static  Fields  (Globals) 

Sometimes  we  need  to  associate  permissions  with  static  fields.  (Static  fields  arc  similar  to  global  variables 
in  procedural  languages.)  Plural  currently  simply  allows  doing  so  with  permission  annotations  directly  on 
the  field.  To  simplify  matters,  however,  static  fields  can  only  be  associated  with  permissions  that  can  be 
duplicated  (more  precisely,  split  into  equally  powerful  permissions,  i.e.  share,  immutable,  and  pure).  This 
avoids  problems  when  permissions  from  a  static  field  access  are  still  in  use  when  the  same  static  field  is 
accessed  a  second  time. 

In  order  to  allow  unique  and  full  permissions  for  static  fields,  one  could  treat  static  fields  as  fields 
of  the  suiTounding  “class”  object,  which  would  require  passing  around  permissions  for  the  surrounding 
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“class”  objects  to  where  their  static  fields  arc  accessed.  Another  option  would  be  a  simple  effect  system  that 
indicates  which  static  fields  arc  accessed  in  the  dynamic  scope  of  a  method. 

6.5.4  Arrays 

Plural  associates  arrays  with  a  permission  of  their  own  and  makes  sure  that  a  modifying  permission  (unique, 
full,  or  share)  is  available  upon  stores  into  the  array.  Plural  does  not  currently  allow  tracking  permissions 
for  the  array  elements,  which  is  subject  of  future  work.  This  issue  is  related  to  putting  permissions  into 
containers  such  as  lists  and  sets. 

6.5.5  Future  Work 

This  section  discusses  how  additional  common  language  features  could  be  treated.  Plural,  mostly  thanks  to 
Crystal,  currently  will  not  crash  when  encountering  these  features,  but  it  also  will  not  do  anything  special 
for  handling  them. 

Inner  Classes.  Non-static  inner  classes  in  Java  implicitly  carry  a  (final)  reference  to  an  object  of  the 
surrounding  class.  (Local  and  anonymous  inner)  classes  defined  inside  method  bodies  also  implicitly  hold 
references  to  (final)  local  variables  defined  in  the  surrounding  method.  Plural  is  currently  unable  to  associate 
permissions  with  these  implicit  fields,  but  it  appeal's  that  this  is  mostly  a  syntactic  problem.  Furthermore,  we 
note  that  permission  parameters  (see  Section  6.4.5)  should  be  a  good  fit  for  these  fields,  since  they  cannot 
be  re-assigned.  Finally,  Eclipse  offers  refactorings  to  promote  inner  classes  to  regular  classes,  making  the 
implicit  fields  explicit.  Plural  can  handle  the  resulting  regular  classes  with  no  problem. 

Exceptions.  Exceptions  to  a  first  approximation  just  induce  exceptional  control  flow  paths  through  meth¬ 
ods.  Crystal  models  control  flow  paths  of  “checked”  exceptions  and  therefore  correctly  propagates  Plural’s 
permission  information  along  exceptional  paths.  However,  this  assumes  that  the  declared  post-condition  of 
methods  hold  even  if  exceptions  are  thrown.  This  may  not  always  be  the  case,  as  was  previously  discovered 
(Naeem  and  Lhotak,  2008).  Plural  could  realistically  be  extended  to  support  annotations  similar  to  dynamic 
state  tests  annotations:  for  example,  an  exception  could  indicate  that  the  receiver  or  parameter  is  in  a  cer¬ 
tain  state.  This  information  could  then  be  propagated  only  on  exceptional  branches,  which  Crystal’s  branch 
sensitivity  allows  us  to  do. 


6.6  Use  Cases 

Plural  was  designed  to  help  developers  cope  with  API  protocols.  This  section  lists  specific  use  cases  for 
using  Plural  in  the  different  development  phases. 

•  Document  and  understand  interface  protocols.  Plural’s  annotations  can  be  used  to  formally  document 
API  protocols.  Developers  will  be  able  to  understand  these  protocols  without  extensively  studying  the 
informal  API  documentation.13 

13Ongoing  efforts  aim  at  visualizing  API  protocols  for  more  convenient  inspection  by  developers. 
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•  Support  development.  Plural  can  be  used  while  developing  code  to  ensure  that  it  conforms  to  protocols 
of  interest;  both  when  developing  clients  and  implementations  of  an  API.  Since  the  tool  can  be  run  on 
a  single  compilation  unit  at  a  time,  developers  can  frequently  re-check  the  code  they  are  working  on. 

•  Retroactively  check  interface  protocols.  It  is  also  possible  to  check  protocols  in  existing  codebases, 
which  is  what  we  did  in  our  case  studies  (see  Chapter  7). 

•  Encode  heap  shape  and  access  patterns.  Even  when  protocols  are  not  of  interest,  developers  can  use 
permissions  to  encode  heap  shape  and  access  patterns.  In  particular,  permissions  can  enforce  owner¬ 
ship  of  objects  (with  unique  or  full  permissions)  and  read-only  access  to  objects  (with  immutable  and 
pure  permissions. 

•  Facilitate  maintenance.  Plural  supports  software  evolution  because  it  can  re-check  changed  code 
based  on  the  persistent  developer  annotations  to  make  sure  that  protocols  arc  still  respected  after  a 
maintenance  task. 
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Chapter  7 

Evaluation 


This  chapter  discusses  a  number  of  case  studies  that  we  performed  based  on  Plural,  the  prototype  protocol 
checking  tool  described  in  the  previous  chapter.  These  case  studies  evaluate  protocol  specification  and 
checking: 

•  Specification  of  several  Java  standard  APIs  using  Plural’s  annotations  (Section  7.1). 

•  Checking  of  compliance  to  the  specified  APIs  in  two  open-source  codebases  (Sections  7.2  and  7.3). 

Section  7.4  discusses  lessons  learned  from  these  case  studies.  Earlier  versions  of  some  of  the  material 
presented  in  this  chapter  is  to  appeal-  at  the  ECOOP  2009  conference  (Bierhoff  et  al.,  2009). 


7.1  Specifying  APIs 

This  section  summarizes  our  experience  specifying  Java  Database  Connectivity  (JDBC),  Collections,  and 
several  other  APIs  that  are  part  of  the  Java  standard  library. 

7.1.1  Java  Database  Connectivity 

The  Java  Database  Connectivity  (JDBC)  API  defines  a  set  of  interfaces  that  Java  programs  can  use  to  access 
relational  databases  with  SQL  commands.  Database  client  applications  access  databases  primarily  through 
Connection,  Statement,  and  ResultSet  objects.  Clients  first  acquire  a  Connection  which  typically 
requires  credentials  such  as  a  username  and  password.  Then  clients  can  create  an  arbitrary  number  of 
Statements  on  a  given  connection.  Statements  are  used  to  send  SQL  commands  through  the  connection. 
Query  results  are  returned  as  ResultSet  objects  to  the  client.  By  convention,  only  one  result  set  can  be  open 
for  a  given  statement;  sending  another  SQL  command  “implicitly  closes”  or  invalidates  any  existing  result 
sets  for  that  statement.  Database  vendors  provide  database-specific  implementations  of  these  interfaces. 

This  section  discusses  the  specification  of  the  major  interfaces  (including  subtypes)  using  Plural  anno¬ 
tations.  The  specified  interfaces  are  massive:  they  define  440  methods,  each  of  which  is  associated  with 
about  20  lines  of  informal  documentation  in  the  source  files  themselves,  for  a  total  of  almost  10,000  lines 
including  documentation  (see  Table  7.1). 
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JDBC 

interface 

Lines 

(Increase) 

Methods 

On 

methods 

State 

space 

Total 

Mult. 

cases 

Connection 

1259 

(9.8%) 

47 

84 

4 

88 

2 

Statement 

936 

(9.4%) 

40 

64 

2 

66 

0 

PreparedStatement 

1193 

(5.5%) 

55 

58 

0 

58 

0 

CallableStatement 

2421 

(5.0%) 

111 

134 

1 

135 

0 

ResultSet 

4057 

(15.4%) 

187 

483 

8 

491 

82 

Total 

9866 

(10.4%) 

440 

823 

15 

838 

84 

Table  7.1:  Specified  JDBC  interfaces  with  total  lines,  size  increase  due  to  annotations,  methods,  annotation 
counts  (on  methods,  for  defining  state  spaces,  and  total),  and  the  use  of  multiple  method  cases  in  each  file. 
The  files  length  is  almost  entirely  due  to  extensive  informal  documentation. 

@States({"open",  "closed"}) 

public  interface  Connection  { 

@Capture(param  =  conn") 

@Perm(requires  =  "share(this,  open)",  ensures  =  'unique(result)  in  open") 

Statement  createStatement()  throws  SQLException; 

@Full(ensures  =  "closed") 

void  close()  throws  SQLException; 

@Pure 

@TmeIndicates("closed") 

boolean  isClosedt)  throws  SQLException;  } 

Figure  7.1:  Simplified  JDBC  Connection  interface  specification.  Creating  a  statement  captures  a  connec¬ 
tion  permission. 


Connections 

The  Connection  interface  primarily  consists  of  methods  to  create  statements,  to  control  transactional 
boundaries,  and  a  close  method  to  disconnect  from  the  database  (Figure  7.1).  Closing  a  connection  inval¬ 
idates  all  statements  created  with  it,  which  will  lead  to  runtime  errors  when  using  an  invalidated  statement. 
Due  to  space  limits,  we  do  not  discuss  our  specification  of  transaction-related  features  here,  but  they  are 
included  in  Table  7.1. 

Our  goal  was  to  specify  JDBC  in  such  a  way  that  statements  and  result  sets  are  invalidated  when  their 
connections  are  closed.  Our  solution  is  a  variant  on  our  work  with  iterators  (Section  3.1):  we  capture  a 
share  connection  permission  each  time  a  statement  is  created  on  it.  The  captured  permission  has  the  open 
state  guarantee,  which  guarantees  that  the  connection  cannot  be  closed  while  the  statement  is  active.  Plural 
releases  the  captured  connection  permission  from  a  statement  that  is  no  longer  used  or  when  the  statement 
is  closed,  as  explained  in  Section  6.4.5.  When  all  statements  are  closed  then  a  full  permission  for  the 
connection  can  be  re-established,  allowing  close  to  be  called. 
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@Retine({ 

@States({"open",  "closed"}), 

@States(refined  =  "open",  value  =  { "hasResultSet",  "noResultSet"},  dim  =  "rs")  }) 

@Param(name  =  "conn",  type  =  Connection. class,  releasedFrom  =  "open") 

public  interface  Statement  { 

@Capture(param  =  stmt") 

@Perm(requires  =  "full(this,  open)",  ensures  =  'unique(result)  in  scrolling") 

ResultSet  executeQuery(String  sql)  throws  SQLException; 

@Share("open") 

int  executeUpdatefString  sql)  throws  SQLException; 

@Full("open") 

@TrueIndicates("hasResultSet") 

@FalseIndicates("noResultSet") 

boolean  execute(String  sql)  throws  SQLException; 

@Capture(param  =  stmt") 

@Perm(requires  =  "full(this,  open)  in  hasResultSet”,  ensures  =  "unique(result)  in  scrolling") 

ResultSet  getResultSet()  throws  SQLException; 

@Full("open") 

@TrueIndicates("hasResultSet") 

@FalseIndicates("noResultSet") 

boolean  getMoreResults()  throws  SQLException; 

@Full(ensures  =  "closed") 

@Release("conn") 
void  close();  } 

Figure  7.2:  JDBC  Statement  interface  specification  (fragment).  The  captured  connection  parameter  “conn” 
is  released  when  the  statement  is  closed  or  garbage-collected. 


Statements 

Statements  are  used  to  execute  SQL  commands.  Statements  define  methods  for  running  queries,  updates, 
and  arbitrary  SQL  commands  (Figure  7.2). 

We  specify  executeQuery  similarly  to  how  statements  are  created  on  connections.  The  resulting 
ResultSet  object  captures  a  full  permission  to  the  statement,  which  enforces  the  requirement  that  only 
one  result  set  per  statement  exists.  Conversely,  executeUpdate  borrows  a  share  statement  permission  and 
returns  the  number  of  updated  rows.  Since  share  and  full  permissions  cannot  exist  at  the  same  time,  result 
sets  have  to  be  closed  before  calling  executeUpdate.  The  Statement  documentation  implies  that  result 
sets  should  be  closed  before  an  update  command  is  run,  and  our  specification  makes  this  point  precise. 

The  method  execute  can  run  any  SQL  command.  If  it  returns  true  then  the  executed  command  was 
a  query,  which  we  indicate  with  the  state  hasResultSet.  getResultSet  requires  this  state  and  returns  the 
actual  query  result. 
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In  rare  cases  a  command  can  have  multiple  results,  and  getMoreResults  advances  to  the  next  result. 
Again,  true  indicates  the  presence  of  a  result  set.  We  use  a  full  permission  because,  like  execute  methods, 
getMoreResults  closes  any  active  result  sets,  as  stated  in  that  method’s  documentation:  “Moves  to  this 
Statement  object’s  next  result,  returns  true  if  it  is  a  ResultSet  object,  and  implicitly  closes  any  current 
ResultSet  object(s)  obtained  with  the  method  getResultSet.” 

Besides  a  plain  Statement  interface  for  sending  SQL  strings  to  the  database,  JDBC  defines  two  other 
flavors  of  statements,  “prepared”  and  “callable”  statements.  The  former  correspond  to  patterns  into  which 
parameters  can  be  inserted,  such  as  search  strings.  The  latter  correspond  to  stored  procedures. 

Since  these  interfaces  arc  subtypes  of  Statement  they  inherit  the  states  defined  for  Statement.  The 
additional  methods  for  prepared  statements  arc  straightforward  to  define  with  these  states,  while  callable 
statements  need  an  additional  state  distinction  for  detecting  NULL  cell  values  (similar  to  Re  suit  Set’s 
wasNull,  see  below). 

Overall,  we  were  surprised  at  how  well  our  approach  can  capture  the  design  of  the  Statement  interfaces. 


Result  sets 

ResultSet  is  the  most  complex  interface  we  encountered.  We  already  discussed  its  most  commonly  used 
features  in  Chapter  1.  In  addition,  result  sets  allow  for  random  access  of  their  rows,  a  feature  that  is  known  as 
“scrolling”.  Scrolling  caused  us  to  add  a  begin  state  besides  valid  and  end,  which  represents  the  result  set’s 
cursor  pointing  “before”  the  first  row.  This  allows  specifying  the  backwards-scrolling  method  previous 
analogously  to  next. 

Furthermore,  the  cell  values  of  the  current  row  can  be  updated,  which  caused  us  to  add  orthogonal 
substates  inside  valid  to  keep  track  of  whether  there  is  a  pending  update  (in  parallel  to  read  and  unread,  see 
Figure  6.1). 

Finally,  result  sets  have  a  buffer,  the  “insert  row”,  for  constructing  a  new  row.  The  problem  is  that,  quot¬ 
ing  from  the  documentation  for  moveToInsertRow,  “[ojnly  the  updater,  getter,  and  insertRow  methods 
may  be  called  when  the  cursor  is  on  the  insert  row.”  Thus,  scrolling  methods  (such  as  next)  arc  not  avail¬ 
able  while  on  the  insert  row,  although  the  documentation  for  these  methods  does  not  hint  at  this  problem. 

Our  interpretation  is  to  give  result  sets  two  modes  (i.e.,  states),  scrolling  and  inserting,  where  the  for¬ 
mer  contains  the  states  for  scrolling  (shaded  in  Figure  1.1)  as  substates.  moveToInsertRow  and  moveTo- 
CurrentRow  switch  between  these  modes  (Figure  6.1).  In  order  to  make  the  methods  for  updating  cells 
applicable  in  both  modes  we  use  method  cases  (illustrated  for  the  method  updatelnt  in  Figure  6.1)  which 
account  for  all  82  methods  with  multiple  cases  in  ResultSet  (see  Table  7.1). 

Figure  7.3  shows  a  fragment  of  the  ResultSet  interface  with  our  actual  protocol  annotations.  Notice 
how  the  two  modes  affect  the  methods  previously  shown  in  Figure  6.1.  The  figure  also  shows  selected 
methods  for  scrolling,  updating  (including  method  cases),  and  inserting.1 

7.1.2  Java  Collections  Framework 

The  Java  Collections  Framework  can  be  seen  as  consisting  of  two  parts: 

’updatelnt  defines  two  cases,  which  are  both  based  on  a  borrowed  full  permission.  One  case  requires  that  permission  in  the 
valid  state  and  ensures  pending ,  while  the  other  case  requires  and  ensures  insert. 
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@Refine({ 

@States({"open",  "closed"}), 

@States(refined  =  "open",  value  =  {"scrolling",  "inserting"}), 

@States(refined  =  "scrolling",  value  =  {"start",  "valid",  "end"},  dim  =  "row"), 

@States(refined  =  "valid",  value  =  {"read",  "unread"},  dim  =  "access"), 

@States(refined  =  "valid",  value  =  {"noUpdates",  "pending"},  dim  =  "update"), 

@States(refined  =  "open",  value  =  { "forwardOnly",  "scrollable" },  dim  =  "scroll",  marker  =  true), 
@States(refined  =  "open",  value  =  {"readonly",  "updatable"},  dim  =  cursor",  marker  =  true)  }) 
@Param(name  =  "stmt",  type  =  Statement. class,  releasedFrom  =  "open") 
public  interface  ResultSet  { 

//  changes  from  figure  6.1 

@Full(guarantee  =  "scrolling",  requires  =  "noUpdates") 

@TrueIndicates("noUpdates") 
boolean  next()  throws  SQLException; 

@Full(guarantee  =  "scrolling",  requires  =  "valid",  ensures  =  'read") 
int  getlnt(int  columnlndex)  throws  SQLException; 

@Pure(guarantee  =  "scrolling",  requires  =  read",  ensures  =  "read") 
boolean  wasNull()  throws  SQLException; 

//  sample  additional  methods 

@Pure("open") 

@TrueIndicates("begin") 

boolean  isBeforeFirst()  throws  SQLException; 

@Full(guarantee  =  "open",  requires  =  "updatable") 

@Cases({ 

@Perm(requires  =  "this  in  valid",  ensures  =  "this  in  pending"), 

@Perm(requires  =  "this  in  insert",  ensures  =  "this  in  insert") 

}) 

void  updatelnt(int  columnlndex,  int  x)  throws  SQLException; 

@Full(guarantee  =  "scrolling",  requires  =  "pending",  ensures  =  "noUpdates") 
void  updateRow()  throws  SQLException; 

@Full(guarantee  =  "open",  ensures  =  "inserting") 
void  moveToInsertRow()  throws  SQLException;  } 

Figure  7.3:  JDBC  ResultSet  interface  specification  (fragment). 
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•  Interfaces  defining  APIs  for  common  object  containers,  including  lists,  sets,  maps,  and  queues,  as 
well  as  the  Iterator  interface  for  iterating  over  those. 

•  Standard  implementations  of  these  interfaces  with  common  data  structures  such  as  arrays,  linked  lists, 
hashtables,  and  trees. 

We  focused  on  specifying  the  interfaces.  Following  the  naming  convention  in  the  API,  we  will  use 
the  term  “collection”  to  include  lists,  sets,  and  queues,  i.e.,  anything  that  can  be  iterated.  Maps  cannot  be 
iterated  directly;  instead,  their  key-,  value-,  and  entry-sets  can  be  iterated  (see  below).  We  will  use  the  term 
“container”  to  include  collections  and  maps. 

Many  researchers,  including  the  author,  have  used  iterators  as  a  case  study  for  their  specification  and 
verification  approaches  (Died  and  Muller,  2005;  Bierhoff,  2006;  Krishnaswami,  2006;  Haack  and  Hurlin, 
2008,  is  an  incomplete  list),  although  most  of  these  study  only  some  aspects  of  the  iterator  protocol  and  do 
not  necessarily  define  iterators  in  the  same  way  as  the  Java  standard  library  API. 

While  the  protocol  for  using  iterators  to  retrieve  elements  from  the  iterated  collection  is  comparably 
simple  (Figure  3.1),  the  challenge  that  most  researchers  have  traditionally  focused  on  is  that  of  concurrent 
modification',  while  a  collection  is  iterated,  it  must  not  be  modified.  Section  3.1  discusses  the  iterator 
protocol  in  more  detail;  Figure  7.8  shows  a  simplified  specification  with  Plural  annotations. 

Faithful  to  the  Java  standard  library,  “read-only”  operations  that  access  the  collection  directly  arc  permit¬ 
ted  in  our  approach  (Bierhoff,  2006),  as  does  Krishnaswami  (2006).  On  the  other  hand,  Haack  and  Hurlin 
(2008)  do  not  seem  to  allow  accessing  an  iterated  collection  at  all. 

Studying  the  Java  Collections  API  reveals  a  number  of  additional  challenges. 

•  Views.  The  API  defines  a  variety  of  operations  for  creating  “views”  on  a  given  container.  In  particular, 
the  Map  interface  provides  methods  keySet,  values,  and  entrySet  that  return  Sets  of  keys,  values, 
and  Map .  Entry  objects.  The  latter  represent  the  key-value  pairs  stored  in  a  map.  One  way  of  looking 
at  iterators  is  to  consider  them  as  another  kind  of  view  (one  with  extremely  limited  interface). 

•  Modifiability.  Some  containers  cannot  be  modified,  i.e.,  elements  cannot  be  added  or  removed.  In 
particular,  the  Java  library's  emptySet,  emptyMap,  and  emptyList  methods  return  empty  containers 
to  which  no  elements  can  be  added.  Views  and  iterators  can  also  be  unmodifiable  in  order  to  avoid 
concurrent  modification:  while  a  collection  is  iterated,  its  views  should  not  be  used  to  modify  the 
collection.  And  if  multiple  iterators  over  the  same  collection  exist  then  none  of  these  iterators  should 
modify  the  collection,  while  a  single  iterator  is  allowed  to  remove  elements  from  the  collection  being 
iterated. 

When  using  access  permissions  it  makes  sense  to  consider  concurrent  modification,  views,  and  modifia¬ 
bility  at  the  same  time.  In  particular,  we  used  the  following  intuition  to  specify  Java  collections  (Figure  7.4 
shows  these  ideas  at  work  to  specify  the  creation  of  iterators  over  collections): 

•  Views  including  iterators  “capture”  a  permission  for  the  underlying  container.  When  views  and  iter¬ 
ators  are  dead  then  the  captured  permission  can  be  recovered  (Section  6.4.5).  For  instance,  iterators 
capture  a  permission  for  the  iterated  over  (“underlying”)  collection. 

•  If  the  view  or  iterator  wants  to  modify  the  container  it  will  capture  a  full  permission  (this  could  be 
relaxed  to  a  share  permission). 
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public  interface  Iterable<T>  { 

@Capture(param  =  "underlying") 

@Cases({ 

@Perm(requires  =  "full(this)",  ensures  =  "unique(result)  in  modifiable"), 
@Perm(requires  =  "immutable(this)",  ensures  =  'unique(result)  in  readonly") 

}) 

Iterator<T>  iterator!); 


Figure  7.4:  Method  cases  allow  iterators  to  be  read-only  or  modifiable  at  the  client’s  choice 


•  Read-only  (unmodifiable)  views  and  iterators  will  capture  an  immutable  permission  for  the  container. 

•  Method  cases  “delay”  the  choice  between  modifiable  and  read-only  views  until  the  client  forces  a 
decision  by  modifying  the  view  or  creating  a  second  view  (forcing  both  to  be  immutable).2  For 
instance,  creating  an  iterator  can  be  defined  with  two  cases,  one  capturing  a  full  and  one  capturing  an 
immutable  collection  permission.  If  a  full  collection  permission  is  available  when  iterator  is  called 
then  both  options  will  be  earned  forward  as  possible  choices  until  one  of  them  becomes  inconsistent 
(see  Section  6.4.3). 

•  Modifiability  for  containers  and  views  is  distinguished  using  two  marker  states  (see  Section  6.4.4), 
“modifiable”  and  “readonly”,  as  used  in  the  two  method  cases  in  Figure  7.4. 

These  rules  reveal  an  interesting  interplay  between  states  and  permissions  for  encoding  modifiability. 
A  modifying  permission  such  as  full  does  not  guarantee  that  the  referenced  view  or  iterator  can  be  used  to 
modify  a  container:  the  permission  also  needs  the  “modifiable”  typestate.  This  typestate  will  imply  that  the 
view  or  iterator  has  a  full  permission  for  the  underlying  container  (with  “modifiable”  typestate),  which  is 
needed  to  modify  the  underlying  collection. 

Deep  Collections 

Our  work  with  PMD  (Section  7.3)  motivated  a  closer  look  at  collection  nesting.  PMD  frequently  uses 
containers  as  elements  in  other  containers,  such  as  lists  of  lists  or  maps  of  sets.  In  these  cases,  we  need  to 
track  permissions  for  the  elements  of  the  outer  container  in  order  to  access  the  inner  one. 

Similar  to  Haack  and  Hurlin  (2008),  we  can  do  so  by  distinguishing  “shallow”  and  “deep”  containers 
(again  using  marker  states).  Deep  containers  hold  permissions  for  their  elements,  while  shallow  ones  do  not 
hold  on  to  permissions  for  their  elements.  In  general,  one  could  want  to  store  any  kind  of  permission  for  the 
elements  of  a  container,  but  in  PMD  we  were  mostly  interested  in  keeping  unique  permissions  for  “shallow” 
containers  in  a  “deep”  container. 

This  idea  has  important  consequences:  whenever  elements  are  added  to  a  deep  container,  the  appropriate 
permission  for  the  new  element  must  be  available.  (Figure  7.5  illustrates  this  for  a  list  that  keeps  unique 
permissions  when  it  is  deep.)  More  problematic,  however,  is  how  to  access  the  elements  in  a  container. 

2We  originally  presented  this  idea  in  Bierhoff  (2006). 
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@States(value  =  {"shallow",  "deep"},  marker  =  true) 
public  interface  List<T>  { 

@Full 

@Cases({ 

@Perm(requires  =  "this  in  shallow"), 

@Perm(requires  =  "this  in  deep  *  unique(#0)") 

}) 

public  boolean  add(T  e); 

@Lend 

@Cases({ 

@Perm(requires  =  "pure(this)  in  shallow"), 

@Perm(requires  =  "unique(this)  in  deep",  ensures  =  "unique(result)"), 
@Perm(requires  =  "immutable(this)  in  deep",  ensures  =  "immutable(result)") 

}) 

public  T  get(int  index); 

} 


Figure  7.5:  List  interface  that  optionally  maintains  permissions  for  contained  elements.  “#0”  in  ©Perm 
annotations  refers  to  the  first  argument  of  the  annotated  method 


A  method  such  as  get  (for  retrieving  an  element  from  a  list  or  map)  should  return,  for  deep  containers,  a 
permission  for  the  contained  object  so  clients  can  call  methods  on  that  object. 

But  now  the  container  invariant  is  in  violated:  while  the  client  is  working  with  the  element  returned  from 
get,  the  container  does  not  have  the  permission  for  this  element.3  What  has  to  happen,  conceptually,  is  that 
the  element’s  permission  is  returned  to  the  container  before  the  container  is  used  again.  We  can  enforce  this 
by  boiTowing  the  element  permission  “from”  the  container  (Boyland  et  ah,  2007),  temporarily  consuming 
the  container  permission  until  the  element  is  no  longer  used  (with  the  ©Lend  annotation  in  Figure  7.5).  This 
is  similar  to  how  we  capture  a  container  permission  in  its  view  and  release  it  when  the  view  is  no  longer 
used,  and  in  fact  the  implementation  in  Plural  is  identical.  As  detailed  in  Boyland  et  al.  (2007),  this  is  legal 
because  the  lender,  the  container,  remains  unpacked  until  the  lending  is  over,  which  allows  it  to  violate  its 
invariant4. 

But  what  if  we  borrow  an  element  from  a  deep  container  for  which  we  have  an  immutable  permission? 
Surely  we  cannot  expect  a  unique  permission  for  the  borrowed  element,  since  other  permissions  might 
be  used  to  borrow  the  very  same  element.  In  this  case,  we  will  only  get  a  immutable  permission  for  the 
borrowed  element  as  well  (Figure  7.5).  This,  in  particular,  also  applies  to  “readonly”  views  and  iterators. 

We  heavily  use  method  cases  for  specifying  container  methods.  They  can  express  the  different  behavior 
of  many  container  methods  depending  on  whether  a  unique  or  immutable  permission  is  available  (e.g„  they 
create  views  in  different  states  and  return  different  permissions  for  contained  elements). 

Outlook.  While  we  limited  our  discussion  here  to  two  kinds  of  collections,  shallow  and  deep,  there  is 
no  fundamental  reason  why  we  could  not  use  more  method  cases  (and  corresponding  marker  states)  to  put 

3This  appears  to  only  be  an  issue  with  unique  and  full  permissions;  the  container  could  split  immutable,  share,  and  pure 
permissions  into  two,  retain  one  of  them,  and  give  the  other  one  to  the  client. 

4Unpacking  is  equivalent  to  what  is  called  “carving”  in  Boyland  et  al.  (2007). 
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other  permissions  for  each  element  into  a  container.  In  fact,  we  did  so  for  PMD.  There,  we  distinguish 
“quietElems”  collections  besides  shallow  and  deep  ones,  which  contain  full  permissions  in  the  “quiet”  state. 
This  is,  of  course,  far  less  convenient  than  being  able  to  “parameterize”  a  container  with  the  permission 
it  keeps  for  each  of  its  elements,  and  our  experience  here  suggests  that  such  a  parametrization  is  in  fact 
straightforward,  although  not  supported  by  Plural  at  this  time. 

Although  not  pursued  in  this  dissertation,  we  believe  that  the  idea  of  “deep”  collections  can  be  adapted 
for  tracking  permissions  for  array  elements. 

7.1.3  Other  Java  Standard  Libraries 

Regular  expressions.  The  API  includes  two  classes.  A  Pattern  is  created  based  on  a  given  regular 
expression  string.  Then,  clients  call  find  or  match  to  match  the  pattern  in  a  given  string,  both  of  which 
return  an  instance  of  Matcher.  The  Matcher  instance  returned  by  these  operations  can  be  used  to  retrieve 
details  about  the  current  match  and  to  find  the  next  matching  substring. 

We  easily  specified  this  protocol  in  Plural.  As  with  iterators,  we  capture  a  immutable  Pattern  permis¬ 
sion  in  each  Matcher.  We  use  a  typestate  matched  to  express  a  successful  match  and  require  it  in  methods 
that  provide  details  about  the  last  match. 

Streams.  Streams  are  discussed  in  detail  in  Section  3.2.  Their  protocol  is  straightforward  to  specify  with 
Plural's  annotations.  We  point  out  that  “buffered”  streams  and  other  kinds  of  wrappers  end  up  “capturing” 
a  permission  for  the  wrapped  stream.  There  should  only  be  one  wrapper  per  stream  (which  we  can  enforce 
by  capturing  a  full  permission  in  them),  but  wrappers  are  nested  inside  one  another  in  practice,  which  is  no 
problem  for  our  approach. 

Exceptions.  When  creating  an  exception,  a  “cause”  (another  exception)  can  be  set  once,  either  using  an 
appropriate  constructor  or,  to  our  surprise,  using  the  method  initCause.  The  latter  retrofits  causes  into 
legacy  exceptions  defined  before  exception  causes  were  introduced  in  Java  1.4  (Figure  7.6).  This  protocol 
is  trivial  to  specify  in  Plural,  but  it  was  fascinating  that  even  something  as  simple  as  an  exception  has  a 
protocol. 

7.2  Beehive:  Verifying  an  Intermediary  Library 

This  section  summarizes  a  case  study  in  using  Plural  for  checking  API  compliance  in  a  third-party  open 
source  codebase,  Apache  Beehive. 

Beehive5  is  an  open-source  library  for  declarative  resource  access.  We  have  focused  on  the  paid  of 
Beehive  that  accesses  relational  databases  using  JDBC.  Beehive  clients  define  Java  interfaces  and  use  Java 
annotations  to  associate  the  SQL  command  to  be  run  when  methods  in  these  interfaces  are  called.  Notice 
that  this  design  is  highly  generic:  the  client-specified  SQL  commands  can  include  parameters  that  are  filled 
with  the  parameters  passed  to  the  associated  method.  Beehive  then  generates  code  stubs  implementing  the 
client-defined  interfaces  that  simply  call  a  generically  written  SQL  execution  engine,  JdbcControl,  whose 
implementation  we  discuss  below. 

3http : / /beehive . apache . org/ 
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/** 

*  Initializes  the  <i>cause</i>  of  this  throwable  to  the  specified  value. 

*  (The  cause  is  the  throwable  that  caused  this  throwable  to  get  thrown.) 

* 

*  <p>This  method  can  be  called  at  most  once.  It  is  generally  called  from 

*  within  the  constructor,  or  immediately  after  creating  the 

*  throwable.  If  this  throwable  was  created 

*  with  {  @  link  #Throwable(Throwable )j  or 

*  l  @  link  #Tlirowable(  String,  Throwable) },  this  method  cannot  be  called 

*  even  once. 

* 

*  @param  cause  the  cause  (which  is  saved  for  later  retrieval  by  the 

*  l@link#getCause()j  method).  (A  <tt>null</tt>  value  is 

*  permitted,  and  indicates  that  the  cause  is  nonexistent  or 

*  unknown.) 

*  @return  a  reference  to  this  <code>Throwable</code>  instance. 

*  @ throws  IllegalArgumentException  if  <code>cause</code>  is  this 

*  throwable.  (A  throwable  cannot  be  its  own  cause.) 

*  @  throws  Illegal StateException  if  this  throwable  was 

*  created  with  { @link  #Throwable(Throwable)j  or 

*  l  @ link  #Tlirowable( String,  Throwable)!,  or  this  method  has  already 

*  been  called  on  this  throwable. 

*  @  since  1.4 
*/ 

public  synchronized  Throwable  initCause(Throwable  cause)  { 
if  (this. cause  !=  this) 

throw  new  IllegalStateException("Can’t  overwrite  cause"); 
if  (cause  ==  this) 

throw  new  IllegalArgumentException("Self— causation  not  permitted"); 
this. cause  =  cause; 

return  this; 


Figure  7.6:  Java  exceptions  have  an  initialization  protocol!  Verbatim  copy  from  the  Java  6  standard  library 
source  code  for  java.lang. Throwable  included  with  JDK  1.6.0_04. 
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We  first  describe  the  APIs  used  by  Beehive  before  discussing  the  challenges  in  checking  that  Beehive 
correctly  implements  a  standard  Java  API  and  its  own  API. 

7.2.1  Checked  Java  Standard  Library  APIs 

We  checked  protocols  of  four  Java  standard  APIs  used  by  Beehive,  highlighting  Plural’s  ability  to  treat  APIs 
orthogonally. 

1.  JDBC.  We  described  the  JDBC  specification  in  Section  7.1.1.  Since  Beehive  has  no  apriori  knowledge 
of  the  SQL  commands  being  executed  (they  are  provided  by  a  client),  it  uses  the  facilities  for  running 
“any”  SQL  command  described  in  Section  7.1.1.  Its  use  of  result  sets  is  limited  to  reading  cell  values, 
and  a  new  statement  is  created  for  every  command.  We  speculate  that  the  Beehive  developers  chose 
this  strategy  in  order  to  ensure  that  result  sets  are  never  rendered  invalid  from  executing  another  SQL 
command,  which  ends  up  helping  our  analysis  confirm  just  that. 

2.  Collections  API.  Beehive  generically  represents  a  query  result  row  as  a  map  from  column  names  to 
values.  One  such  map  is  created  for  each  row  in  a  result  set  and  added  to  a  list  which  is  finally  returned 
to  the  client.  Section  7.1.2  discusses  our  specification  of  maps  and  lists  in  detail. 

3.  Regular  expressions.  Regular  expressions  (see  Section  7.1.3)  are  only  used  once  in  Beehive.  The 
pattern  being  matched  is  a  static  field  in  one  of  Beehive’s  classes,  which  we  annotate  with  @Imm. 

4.  Exceptions.  Beehive  uses  initCause  (see  Figure  7.6)  to  initialize  a  legacy  exception  that  is  paid  of 
the  Java  standard  library,  NoSuchElementException  (which,  incidentally,  is  the  exception  prescribed 
by  the  Java  standard  library  to  signal  a  iterator  protocol  violation).  It  is  unknown  why  the  library 
designers  did  not  retrofit  the  “cause”  mechanism  into  their  own  exceptions,  but  Beehive,  due  to  this 
omission,  has  no  choice  but  to  use  initCause. 

7.2.2  Implementing  an  Iterator 

Beehive  implements  an  Iterator  over  the  rows  of  a  result  set.  Figure  7.7  shows  most  of  the  relevant 
code.  We  use  state  invariants,  i.e.,  predicates  over  the  underlying  result  set  (see  Section  6.3.7),  to  specify 
iterator  states.  Notice  that  alive  is  our  default  state  that  all  objects  are  always  in.  Thus  its  state  invariant  is  a 
conventional  class  invariant  (Leavens  et  al.,  1999;  Barnett  et  al.,  2004)  that  is  established  in  the  constructor 
and  preserved  afterwards. 

When  checking  the  code  as  shown.  Plural  issues  3  warnings  in  hasNext  (see  Table  7.2).  This  is  because 
our  vanilla  iterator  specification  (Bierhoff  and  Aldrich,  2007b)  assumes  hasNext,  which  tests  if  an  element 
can  be  retrieved,  to  be  pure.  Beehive’s  hasNext  is  not  pure  because  it  calls  next  on  the  ResultSet  (Figure 
7.7). 

This  problem  can  be  fixed,  for  example,  by  advancing  the  result  set  to  the  next  row  at  the  end  of  the 
iterator’s  next  method  (after  constructing  the  return  value)  and  remembering  the  outcome  in  the  existing 
flag.  The  iterator  constructor  can  initialize  the  flag  by  moving  the  result  set  to  the  first  row,  if  it  exists. 
That  way,  hasNext  is  pure  because  it  only  tests  whether  the  flag  is  true.  However,  this  code  change  has 
the  disadvantage  that  the  result  set  may  be  unnecessarily  advanced  to  a  row  that  is  never  retrieved  with  a 
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subsequent  call  to  the  iterator’s  next  method.  Furthermore,  it  duplicates  the  code  for  advancing  to  the  next 
row  in  next  and  the  constructor. 

Alternatively,  the  warnings  disappear-  when  we  change  hasNext’s  specification  to  use  a  full  permission. 

Also,  note  that  next’s  specification  requires  available ,  which  guarantees  that  _primed  is  true  (see 
Figure  7.7).  This  makes  the  initial  check  in  next  superfluous  (if  all  iterator  clients  were  checked  with  Plural 
as  well). 

7.2.3  Formalizing  Beehive  Client  Obligations 

Beehive  is  an  intermediary  library  for  handling  resource  access  in  applications:  it  uses  various  APIs  to 
access  these  resources  and  defines  its  own  API  through  which  applications  can  take  advantage  of  Beehive. 
We  believe  that  this  is  a  very  common  situation  in  modern  software  engineering:  application  code  is  arranged 
in  layers,  and  Beehive  represents  one  such  layer.  The  resource  APIs,  such  as  JDBC,  reside  in  the  layer  below, 
while  the  application-specific  code  resides  in  the  layer  above. 

Beehive’s  API  is  defined  in  the  JdbcControl  interface,  which  is  implemented  in  JdbcControlImpl. 
JdbcControlImpl  in  turn  is  a  client  to  the  JDBC  API.  The  class  provides  provides  three  methods  on- 
Acquire,  invoke,  and  onRelease  to  clients.  The  first  one  creates  a  database  connection,  which  the  third 
one  closes,  invoke  executes  an  SQL  command  and,  in  the  case  of  a  query,  maps  the  result  set  into  one 
of  several  possible  representations.  One  representation  is  the  iterator  mentioned  above;  another  one  is  a 
conventional  List.  Each  row  in  the  result  is  individually  mapped  into  a  map  of  key-value  pairs  (one  entry 
for  each  cell  in  the  row)  or  a  Java  object  whose  fields  are  populated  with  values  from  cells  with  matching 
names. 

Notice  that  some  of  these  representations,  notably  the  iterator  representation,  of  a  result  require  the 
underlying  result  set  to  remain  open.  The  challenge  now  is  to  ensure  that  onRelease  is  not  called  while 
these  are  still  in  use  because  closing  the  connection  would  invalidate  the  results.  This  requirement  is  identical 
to  the  one  we  described  for  immediate  clients  of  Connection,  and  thus  we  should  be  able  to  specify  it  in 
the  same  way. 

However,  the  connection  is  in  this  case  a  field  of  a  surrounding  Beehive  JdbcControlImpl  object, 
and  Plural  has  currently  no  facility  for  letting  JdbcControlImpl  clients  keep  track  of  the  permission  for 
one  of  its  fields.  Therefore,  we  currently  work  with  a  simplified  JdbcControlImpl  that  always  closes 
result  sets  at  the  end  of  invoke.  Its  specification,  as  desired,  enforces  that  onAcquire  is  called  before 
onRelease  and  invoke  is  only  called  “in  between”  the  other  two.  This,  however,  means  that  our  simplified 
JdbcControlImpl  does  not  support  returning  iterators  over  result  sets  to  clients,  since  they  would  keep 
result  sets  open.  Overcoming  this  problem  is  discussed  in  the  next  section. 

As  mentioned.  Beehive  generates  code  that  calls  invoke.  The  generated  code  would  presumably  have 
to  impose  usage  rules  similar  to  the  ones  for  invoke  on  its  clients.  Plural  could  then  be  used  to  verify  that 
the  generated  code  follows  JdbcControlImpl ’s  protocol. 

7.2.4  Overhead:  Annotations  in  Beehive 

The  overhead  for  specifying  Beehive  is  summarized  in  Table  7.2.  We  used  about  1  annotation  per  method 
and  5  per  Beehive  class,  for  a  total  of  66  annotations  in  more  than  2,000  lines,  or  about  one  annotation  every 
30  lines.  Running  Plural  on  the  12  specified  Beehive  source  files  takes  about  10  seconds. 
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@ClassStates({ 

@State(name="alive", 

inv="full(_rs, scrolling)  &&  full(_rowMapper)  in  init  &&  _primed  ==  true  =>  _rs  in  valid"), 
@State(name="available",  inv="_primed  ==  true")  }) 

@NonReentrant 

public  class  ResultSetlterator  implements  java.util. Iterator  { 
private  final  ResultSet  _rs; 
private  final  RowMapper  _rowMapper; 
private  boolean  _primed  =  false; 

A*  @  return  true  if  there  is  another  element  */ 

@Pure(guarantee  =  "next",  fieldAccess  =  true) 

@TrueIndicates("available") 

public  boolean  hasNext()  { 
if  (_primed)  { 

return  true; 

} 


try  { 

_primed  =  _rs.next(); 

}  catch  (SQLException  sqle)  { 

return  false; 

} 

return  _primed; 


A*  @return  The  next  element  in  the  iteration.  */ 

@Full(requires  =  available",  ensures  =  "hasCurrent",  fieldAccess  =  true) 
public  Object  next()  { 

try  { 

if  (Lprimed)  { 

_primed  =  _rs.next(); 
if  ( Lprimed)  { 

throw  new  NoSuchElementExceptionQ; 


//  reset  upon  consumption 

_primed  =  false; 

return  _rowMapper.mapRowToReturnType(A  analysis— only  */_rs); 

}  catch  (SQLException  e)  { 

//  Since  Iterator  interface  is  locked ,  all  we  can  do 
//  is  put  the  real  exception  inside  an  expected  one. 

NoSuchElementException  xNoSuch  =  new  NoSuchElementException("ResultSet  exception:  "  +  e); 

xNoSuch.initCause(e); 

throw  xNoSuch; 

} 


Figure  7.7:  Beehive’s  iterator  over  the  rows  of  a  result  set  (constructor  omitted).  Plural  issues  warnings 
because  hasNext  is  impure. 
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Beehive  class 

Lines 

Methods 

Annotations 

Meths.  Invs.  Total 

Plural 

warnings 

False 

pos. 

DefaultlteratorResultSetMapper 

37 

2 

1 

0 

1 

0 

0 

DefaultObjectResultSetMapper 

127 

2 

2 

0 

2 

0 

0 

JdbcControlImpl 

521 

13 

13 

1 

14 

2 

1 

ResultSetHashMap 

85 

9 

9 

0 

9 

0 

0 

ResultSetlterator 

106 

4 

4 

3 

7 

3 

0 

ResultSetMapper 

32 

2 

2 

0 

2 

0 

0 

RowMapper 

260 

5 

9 

1 

10 

0 

0 

RowMapperFactory 

156 

7 

3 

0 

3 

4 

4 

RowToHashMapMapper 

57 

2 

4 

1 

5 

0 

0 

RowToMapMapper 

49 

2 

4 

1 

5 

0 

0 

RowToObjectMapper 

236 

3 

4 

0 

4 

0 

0 

SqlStatement 

511 

14 

4 

0 

4 

0 

0 

Total 

2158 

65 

59 

7 

66 

9 

5 

Table  7.2:  Beehive  classes  checked  with  Plural.  The  middle  paid  of  the  table  shows  annotations  (on  methods, 
invariants,  and  total)  added  to  the  code.  The  last  2  columns  indicate  Plural  warnings  and  false  positives. 

7.2.5  Analysis  Precision 

Plural  reports  9  problems  in  Beehive.  Three  of  them  arc  due  to  the  impure  hasNext  method  in  Result - 
Setlterator  (see  Section  7.2.2).  Letting  hasNext  use  a  full  permission  removes  these  warnings.  Another 
warning  in  JdbcControlImpl  is  caused  by  an  assertion  on  a  field  that  arguably  happens  in  the  wrong 
method:  invoke  asserts  that  the  database  connection  is  open  before  delegating  the  actual  query  execution 
to  another,  “protected”  method  that  uses  the  connection.  Plural  issues  a  warning  because  a  subclass  could 
override  one,  but  not  the  other,  of  these  two  methods,  and  then  the  state  invariants  may  no  longer  be  consis¬ 
tent.  The  warning  disappears  when  moving  the  assertion  into  the  protected  method.  Furthermore  we  note 
that  our  state  invariants  guarantee  that  the  offending  runtime  assertion  succeeds. 

The  remaining  warnings  issued  by  Plural  arc  false  positives.  This  means  that  our  false  positive  rate  is 
is  around  1  per  400  lines  of  code.  We  consider  this  to  be  quite  impressive  for  a  behavioral  verification  tool 
applied  to  complicated  APIs  (JDBC  and  others)  and  a  very  challenging  case  study  subject  (Beehive). 


Sources  of  Imprecision 

The  remaining  warnings  in  Beehive  fall  into  the  following  categories: 

•  Reflection  (1).  Plural  currently  does  not  give  permissions  to  objects  created  using  reflection,  which 
Beehive  uses  in  RowMapperFactory. 

•  Static  fields  (3).  RowMapperFactory  manipulates  a  static  map  object,  which  we  specified  to  re¬ 
quire  full  permissions.  For  soundness,  we  only  allow  duplicable  permissions,  i.e.,  share,  pure,  and 
immutable,  on  static  fields. 
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•  Complex  invariant  ( 1).  JdbcControlImpl  opens  a  new  database  connection  in  onAcquire  only  if 
one  does  not  already  exist.  Plural  currently  cannot  capture  the  invariant  that  a  non-null  field  implies 
a  permission  for  that  field,  which  would  allow  Plural  to  verify  the  code. 


These  arc  common  sources  of  imprecision  in  static  analyses.  We  arc  considering  tracking  fields  as 
implicit  parameters  in  method  calls,  as  discussed  in  Section  7.2.3,  and  static  fields  could  be  handled  in 
this  way  as  well.  Related  to  this  issue  is  also  a  place  in  Beehive  where  a  result  set  that  was  assigned  to 
a  field  in  the  constructor  is  implicitly  passed  in  a  subsequent  method  call.  We  turned  it  into  an  explicit 
method  parameter  for  now  (the  call  to  mapRowToReturnType  in  Figure  7.7).  Java(X)  has  demonstrated 
that  fields  can  be  tracked  individually  (Degen  et  al.,  2007),  although  we  would  like  to  track  permissions 
for  “abstract”  fields  that  do  not  necessarily  correspond  to  actual  fields  in  the  implementation.  We  arc  also 
working  on  a  strategy  for  handling  object  construction  through  reflection,  and  on  generalizing  the  state 
invariants  expressible  in  Plural. 

We  also  simplified  the  Beehive  code  in  a  few  places  where  our  approach  for  tracking  local  aliases  leads  to 
analysis  imprecisions.  Since  local  alias  tracking  is  orthogonal  to  tracking  permissions  we  used  the  simplest 
available,  sound  solution  in  Plural,  which  is  insufficient  in  some  cases.  We  plan  to  evaluate  other  options. 

Problems  occur  when  the  same  variable  is  assigned  different  values  on  different  code  paths,  usually 
depending  on  a  condition.  When  these  code  paths  rejoin.  Plural  assumes  that  the  variable  could  point  to  one 
of  several  locations,  which  forbids  strong  updates.  We  arc  investigating  using  more  sophisticated  approaches 
that  avoid  this  problem.  Alternatively,  Plural  will  work  fine  when  the  paid  of  the  code  that  initializes  a 
variable  on  different  paths  is  refactored  into  a  separate  method.  Notice,  however,  that  tracking  local  aliasing 
is  a  lot  more  tractable  than  tracking  aliasing  globally.  Permissions  reduce  the  problem  of  tracking  aliasing 
globally  to  a  local  problem. 

Furthermore,  we  modified  Beehive  in  one  place  to  not  use  correlated  ifs  (in  a  loop),  which  the  tool 
currently  does  not  support.  This  required  moving  a  dynamic  state  test  and  inserting  a  “break”  statement  into 
a  loop. 

Finally,  we  assumed  one  class  to  be  non-reentrant,  but  we  believe  a  more  complicated  specification 
would  allow  the  class  to  be  analyzed  assuming  reentrancy  (using  intermediate  states  as  seen  in  Section 
3).  Therefore,  we  use  the  (currently  unchecked)  annotation  shown  in  Figure  7.7  to  mark  a  class  as  non¬ 
reentrant,  which  causes  Plural  to  omit  certain  checks  during  API  implementation  checking.  We  arc  planning 
on  checking  this  annotation  with  Plural  in  the  future. 


Refactoring  option 

Notice  that  besides  improving  the  tool  there  is  usually  the  option  of  refactoring  the  problematic  code.  We 
believe  that  this  is  an  indicator  for  the  viability  of  our  approach  in  practice,  independent  of  the  features  sup¬ 
ported  by  our  tool:  developers  can  often  circumvent  tool  shortcomings  with  (fairly  local)  code  changes.  On 
the  other  hand,  we  have  not  seen  many  examples  that  fundamentally  could  not  be  handled  by  our  approach. 
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7.3  Iterators  in  PMD:  Scalability  and  Precision 

We  used  version  3.7  of  PMD  as  it  is  included  in  the  DaCapo  2006-10-MR2  benchmarks6  to  investigate  how 
Plural  can  be  used  to  check  existing  large  codebases.  We  chose  this  codebase  because  it  was  used  as  a 
benchmark  for  recent  whole-program  protocol  analyses  based  on  tracematches  (Bodden  et  ah,  2008;  Naeem 
and  Lhotak,  2008),  which  were  among  others  used  to  check  two  protocols  related  to  iterators  in  PMD.  This 
case  study  investigates  the  use  of  Plural  for  checking  similar  protocols  in  the  same  codebase. 

Unlike  previous  modular  program  verification  tools,  whole-program  analyses  promise  aliasing  flexibil¬ 
ity,  and  tracematch-based  analyses  arc  the  only  ones  we  arc  aware  of  that  can  automatically  check  protocols 
involving  object  dependencies  (such  as  the  “concurrent  modification”  protocol  discussed  below).  There¬ 
fore,  this  case  study  allows  direct  comparison  between  ours  and  the  leading  approaches  with  comparable 
expressiveness. 

In  particular,  we  focused  on  two  well-known  protocol  errors  related  to  iterators  over  collections  (see 
Section  3.1): 

•  Iterator  usage.  It  is  only  legal  to  call  next  on  an  iterator  that  has  another  element  available.  Usually, 
this  requires  calling  and  checking  the  return  value  of  the  iterator’s  hasNext  method,  although  other 
kinds  of  dynamic  state  tests  arc  possible. 

Tracematch-based  analyses  report  high  precision  in  checking  this  protocol,  and  our  case  study  at¬ 
tempts  to  establish  how  much  overhead  our  approach  imposes  when  checking  such  relatively  simple 
protocols  (Section  7.3.1). 

•  Concurrent  modification.  Collections  must  not  be  modified  directly  while  they  are  iterated.  If  collec¬ 
tions  arc  modified  through  an  iterator  (by,  for  instance,  removing  an  element)  then  the  collection  must 
not  be  iterated  over  with  other  iterators.  These  represent  changes  of  a  collection  without  the  knowl¬ 
edge  of  its  iterators,  which  arc  forbidden  because  changes  to  a  collection  may  invalidate  iterators’ 
“pointers”  into  the  collection.  These  errors  arc  called  “concurrent  modification”  even  though  they  can 
easily  occur  in  single-threaded  programs. 

Tracematch-based  analyses  report  low  precision  in  checking  this  protocol  in  PMD,  and  so  we  were 
interested  in  seeing  whether  our  approach  could  provide  added  benefit  (Section  7.3.2). 

After  discussing  our  own  experiences  checking  these  protocols  in  PMD,  Section  7.3.3  compares  our 
result  to  previous  Tracematch-based  analyses. 

Case  studies  with  large  codebases  arc  fundamentally  challenging  to  do  with  a  tool  like  Plural  because 
providing  annotations  for  methods  and  classes  throughout  the  codebase  may  require  considerable  manual 
effort.  This  complicates  case  studies  in  practice.  But  it  also  enables  more  realistic  and  detailed  estimates  of 
the  overhead  on  developers  imposed  by  modular  approaches. 


Background  on  PMD.  PMD  is  a  bug-finding  tool  primarily  for  Java  code.  It  comes  with  a  plethora  of 
“rules”  that  can  be  used  to  expose  common  problems  in  Java  source  files.  Some  of  these  rules  look  for  bugs, 
but  others  arc  targeted  at  enforcing  coding  conventions  such  as  avoiding  large  methods  or  unused  local 

6http : / /dacapobench . org/ 
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variables.  The  version  of  PMD  used  in  this  case  study  consists  of  close  to  40  KLOC  (excluding  comments 
and  blank  lines)  in  almost  450  classes. 

PMD  supports  several  modes  of  use  including  as  an  application  that  can  be  started  from  the  command 
line  to  check  a  given  set  of  rules  in  a  given  set  of  source  tiles,  as  a  “task”  in  an  automated  build  script,  to  run 
“benchmarks”,  and  as  paid  of  a  GUI  application.  A  typical  “run”  consists  of  the  following  steps: 

•  Collect  source  files  and  rules  to  be  checked  against  each  other. 

•  For  each  source  tile: 

-  Parse  the  tile  and  create  a  structured  representation  of  it  as  an  Abstract  Syntax  Tree  (AST). 

-  Check  each  rule,  using  the  AST.  (Most  rules  are  simple  tree  walkers  that  visit  the  AST.) 

•  “Render”  rule  violations  in  the  desired  format,  e.g.,  as  an  HTML  page. 

PMD  makes  heavy  use  of  collections  and  iterations.  Many  collections  are  declared  as  fields  in  another 
class,  while  iterations  typically  happen  locally  to  a  method.  Many  collections  arc  long-lived  in  that  they  arc 
created  close  to  the  staid  of  the  program  and  continue  to  be  used  until  its  end. 

7.3.1  Iterator  Usage  Protocol 

Iterators  arc  widely  used  in  PMD,  and  most  iterations  in  PMD  arc  over  Java  Collections  (see  Section  7.2.1), 
but  PMD  implements  a  few  iterator  classes  over  its  own  data  structures  as  well.  There  arc  170  distinct  call 
sites  invoking  the  next  method  of  the  iterator  interface  in  the  codebase. 

It  was  very  easy  to  check  with  Plural  that  the  protocol  for  using  iterators  was  followed  in  PMD.  We 
simply  used  the  simplified  iterator  specification  shown  in  Figure  7.8  to  ensure  that  PMD 

•  calls  and  tests  hasNext  before  every  call  to  next,  and 

•  calls  remove  at  most  once  after  each  call  to  next. 


Overhead  and  Precision 

Plural  can  verify  that  that  167  out  of  170  calls  to  next  call  and  test  hasNext  first,  remove  is  only  called  in 
one  place,  which  Plural  verifies  as  well. 

It  took  the  author  about  75  minutes  to  check  that  this  protocol  is  followed  in  all  of  PMD,  using  only 
15  annotations.  Most  iterator  usages  could  be  verified  by  Plural  without  any  user  intervention  because  they 
arc  created  and  used  inside  one  method.  Annotations  were  needed  where  iterators  were  returned  from  a 
method  call  inside  PMD  and  then  used  elsewhere.  In  one  place  an  iterator  is  passed  to  a  helper  method  after 
checking  hasNext,  and  we  could  express  the  contract  of  this  helper  method  with  a  suitable  annotation.7 

Three  calls  to  next  could  not  be  verified  in  this  case  study,  although  they  represent  correct  usage:  in 
these  cases,  PMD  checks  that  a  set  is  non-empty  before  creating  an  iterator  over  the  tested  set  and  immedi¬ 
ately  retrieving  (only)  the  first  element.  This  pattern  works  around  a  shortcoming  in  Java’s  Set  interface — the 

7This  method  trips  up  one  of  the  previous  analyses  (Bodden  et  al..  2008). 
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@Refine({ 

@States(value  =  {"available",  "end"},  dim  =  "next"), 

@States(value  =  {"retrieved",  "removed"},  dim  =  "current") 

}) 

public  interface  Iterator<T>  { 

@Pure("next") 

@TrueIndicates("available") 

public  boolean  hasNext(); 

@Full(requires  =  ’available",  ensures  =  "retrieved") 
public  T  next(); 

@Full(guarantee  =  current",  requires  =  "retrieved",  ensures  =  "removed") 
public  void  remove(); 


Figure  7.8:  Simplified  annotations  for  checking  iterator  protocol 


absence  of  a  method  to  get  “some”  element  from  the  set.  (The  only  way  to  get  to  the  elements  of  a  set  is 
to  iterate  over  them.)  We  could  not  verify  these  calls  to  next  because  the  collection  being  iterated  is  not 
considered  in  our  simplified  iterator  protocol.  However,  when  the  collection  is  also  tracked  with  Plural  (see 
below)  then  we  can  verify  this  pattern. 

In  summary,  checking  the  iterator  protocol  in  PMD  using  Plural  was  exceedingly  easy  and  imposed 
almost  no  overhead.  Running  Plural  on  PMD’s  entire  source  tree  of  446  files  (with  the  same  configuration 
as  for  Beehive)  takes  about  4  minutes. 


Iterator  implementations 

PMD  implements  three  iterators  of  its  own.  In  one  of  them,  Treelterator,  the  implementation  of  hasNext 
is  not  only  impure,  like  Beehive’s  iterator,  but  advances  the  iterator  every  time  it  is  called.  Thus,  failure  to 
call  next  after  hasNext  results  in  lost  elements.  The  other  iterators  exhibit  behavior  compatible  with  the 
conceptual  purity  of  hasNext:  next  is  used  to  pre-fetch  the  element  to  be  returned  the  next  time  it  is  called 
before  returning  the  current  element.  hasNext  then  simply  checks  the  pre-fetched  element  is  valid,  which 
is  typically  a  pure  operation. 

In  light  of  these  and  the  iterator  implementation  in  Beehive  (Figure  7.7),  it  appears  legitimate  to  ask 
whether  hasNext  is  really  a  pure  operation.  This  would  have  significant  consequences  for  behavioral  spec¬ 
ification  approaches  like  the  JML  (Leavens  et  al.,  1999)  or  Spec#  (Barnett  et  al.,  2004)  because  they  use 
pure  methods  in  specifications.  Conventionally,  the  specification  of  next  in  the  JML  would  be  “requires 
hasNext  ()”,  but  that  would  be  illegal  if  hasNext  was  not  pure.  In  contrast,  our  specifications  are  more 
robust  to  the  non-purity  of  hasNext.  In  fact,  Plural  can  verify  iterator  usage  in  PMD  with  a  full  permission 
for  hasNext  with  the  same  precision. 
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7.3.2  Concurrent  Modifications 

After  the  initial  success  of  checking  the  “shallow”  iterator  protocol  (see  previous  section)  we  decided  to 
extend  the  PMD  case  study  to  check  for  “concurrent  modifications”.  This  turned  out  to  be  challenging  for 
several  reasons: 

•  Collections  arc  often  held  in  fields,  and  some  of  them  are  long-lived.  Collections  arc  also  often  used 
as  method  parameters  or  return  values.  In  all  these  these  situations  Plural  requires  annotations  in  order 
to  check  most  collection  accesses.  Overall,  PMD  declares  447  lists  and  sets  and  139  maps  as  fields, 
local  variables,  method  parameters,  or  method  return  types. 

•  Additionally,  collections  arc  often  nested:  For  example,  we  found  a  map  from  keys  to  maps  from 
other  keys  to  integers.  Lists  of  lists,  maps  from  keys  to  lists,  etc.,  were  also  very  common.  These  pose 
significant  challenges  that  arc  related  to  tracking  permissions  for  array  cells  (see  Section  7.1.2). 

In  order  to  cope  with  the  many  collections  to  be  tracked  we  limited  the  case  study  to  the  following  core 
parts  of  PMD: 

•  The  handling  of  rules  and  source  files  to  be  analyzed,  including  maps  of  properties  for  configuring 
individual  rules. 

•  The  rules  for  checking  Java  source  files  (thereby  excluding  rules  for  other  types  of  files). 

•  The  handling  and  reporting  of  rule  violations. 

These  represent  many  of  the  collections  being  used  in  a  typical  PMD  run  to  check  Java  source  files 
against  some  set  of  rules,  with  one  important  exception:  we  did  not  attempt  to  check  PMD’s  parsing  and 
binding  infrastructure,  or  the  related  AST  implementation.  This  puts  a  caveat  on  our  checking  of  “rules”,  as 
will  be  discussed  below. 


Overhead 

Table  7.3  summarizes  the  number  of  annotations  used  for  checking  concurrent  modification  problems  in 
PMD,  broken  down  by  top-level  packages.  Notice  that  the  largest  package,  pmd.ast,  is  almost  entirely 
automatically  generated  by  a  parser  generator.  Almost  all  of  the  annotations  in  this  package  arc  on  the  more 
than  100  methods  defined  in  the  AST  visitor  interface  (2  annotations  per  method),  which  we  needed  to 
annotate  in  order  to  be  able  to  track  the  many  collections  stored  in  fields  of  the  various  rules  (most  of  which 
arc  AST  visitors). 

Focusing  on  the  packages  that  we  considered  in  detail  as  paid  of  this  case  study,  we  used  on  average 
about  1  annotations  for  every  two  methods,  or  1  annotation  per  25  lines  of  code.  It  took  the  author  about  1 8 
hours  to  provide  these  annotations,  which  includes  the  time  to  annotate  Java  Collections  classes  as  needed. 
The  code  was  refactored  in  12  places  to  allow  verification  with  Plural,  mostly  to  get  around  limitations  of 
Plural’s  aliasing  analysis  and  inheritance  handling  (see  Section  7.2.5). 
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Considered 

Remaining  warnings 

Package 

impl. 

KLOC 

Annotations 

Initially 

Improved 

pmd 

Yes 

2.2 

160 

13 

13 

pmd.renderers 

Yes 

0.7 

13 

0 

0 

pmd.rules.** 

Yes 

5.4 

156 

58 

33 

pmd.  sourcetypehandlers 

Yes 

0.1 

4 

0 

0 

pmd.stat 

Yes 

0.2 

12 

7 

0 

Considered 

Yes 

8.7 

345 

78 

46 

pmd.  ant 

No 

0.3 

0 

9 

9 

pmd.ast 

No 

14.8 

235 

3 

3 

pmd.cpd.** 

No 

4.2 

7 

41 

41 

pmd.dfa.** 

No 

1.7 

8 

65 

65 

pmd.jaxen 

No 

0.4 

7 

6 

6 

pmd.jsp.** 

No 

6.4 

0 

28 

28 

pmd.  parsers 

No 

0.05 

0 

0 

0 

pmd.quickfix 

No 

0.01 

0 

0 

0 

pmd.  symboltable 

No 

1.1 

8 

52 

52 

pmd.util.** 

No 

1.7 

7 

21 

21 

Not  considered 

No 

30.8 

272 

225 

225 

Total 

Both 

39.4 

617 

303 

271 

Table  7.3:  PMD  annotations  for  preventing  concurrent  modifications.  Package  names  with  **  include  sub¬ 
packages.  The  improvements  to  the  number  of  remaining  warnings  come  from  support  for  permissions 
allowing  dispatch  and  field  access  (Section  6.4.7) 
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Sources  of  Imprecision 

We  are  not  aware  of  any  iterator-related  bugs  in  PMD,  and  so  all  the  remaining  warnings  that  Plural  issues 
have  to  be  considered  false  positives.  While  the  presence  of  false  positives  in  the  parts  of  PMD  whose 
implementation  we  did  not  consider  as  paid  of  the  case  study  (see  Table  7.3)  arc  not  surprising,  this  section 
discusses  in  detail  the  origin  of  the  remaining  warnings  in  the  parts  of  PMD  whose  iterator  and  collection 
usage  we  attempted  to  verify. 

When  we  first  used  Plural  for  checking  PMD  we  did  not  have  support  for  permissions  allowing  both 
dynamic  dispatch  and  field  access  as  described  in  Section  6.4.7.  As  a  result,  42  out  of  78  warnings  (second  to 
last  column  in  Table  7.3)  were  due  to  methods  accessing  fields  and  making  dynamically  dispatched  method 
calls.  This  and  several  other  experiences  motivated  adding  support  for  “dispatch-and-fields”  permissions  to 
Plural  (Section  6.4.7).  As  a  result.  Plural  now  only  issues  46  warnings  in  the  parts  of  PMD  we  considered. 

•  Captured  permission  tracking  (13/46).  Plural  loses  precision  about  permissions  other  than  unique  that 
arc  captured  in  loops,  which  could  be  avoided  with  smarter  lattice  comparison  and  join  implementa¬ 
tions. 

•  Foreign  field  access  (7/46).  One  rule  in  PMD  accesses  fields  of  objects  other  than  the  receiver,  which 
Plural  currently  does  not  support.  We  believe  that  these  direct  field  accesses  could  be  refactored  into 
method  calls. 

•  Unpacking  (5/46).  Unpacking  an  immutable  permission  in  Plural  currently  yields  pure  permissions 
for  fields,  but  this  leads  to  imprecisions  when  immutable  permissions  for  fields  arc  needed.  Plural 
should  yield  immutable  permissions  (as  in  Boyland  et  al.,  2007)  for  fields  associated  with  unique  or 
full  permissions,  which  would  remove  these  warnings. 

•  Unreachable  code  (4/46).  One  Java  rule  violates  method  pre-conditions  in  case  another  method  re¬ 
turns  a  non-null  value,  but  it  appeal's  that  null  is  always  returned.  We  believe  that  these  unreachable 
code  blocks  could  be  detected  and  ignored  by  Plural. 

•  Static  fields  (3/46).  Static  fields  lead  to  problems  similar  to  our  experience  with  Beehive  (see  Section 
7.2.5). 

•  Superclass  fields  (2/46).  Plural  currently  does  not  allow  directly  accessing  superclass  fields  in  sub¬ 
classes.  Unpacking  the  object  to  the  accessed  frame  should  remove  this  problem. 

•  Array,  reflection,  library  call  (3/46).  Array  access,  use  of  reflection,  and  a  call  into  a  XML  library  we 
did  not  specify  cause  one  warning  each. 

•  Miscellaneous  (9/46).  The  remaining  errors  have  various  causes  including  a  place  where  elements  of 
a  “deep”  list  are  moved  into  another  list  one  by  one,  which  our  specification  of  collections  does  not 
support. 

This  discussion  shows  that  the  vast  majority  of  false  positives  come  from  insufficient  support  for  inher¬ 
itance  as  well  as  a  variety  of  tool  shortcomings.  We  believe  that  a  more  mature  tool  could  avoid  most  of  the 
problems  we  found. 
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Handling  the  rest  of  PMD 

As  Table  7.3  suggests,  we  excluded  the  automatically  generated  Java  AST  (package  pmd.ast),  PMD’s  rules 
related  to  JSPs  (pmd.jsp)  and  C++  code  (pmd.cpd),  its  dataflow  analysis  framework  (pmd.dfa),  and  its 
graphical  user  interface  (pmd.util)  from  consideration,  as  well  as  the  already  mentioned  binding  infrastruc¬ 
ture  (pmd.symboltable),  which  in  particular  includes  “scope”  classes  for  maintaining  binding  information. 
Except  for  scopes,  the  excluded  parts  of  PMD  arc  independent  from  the  parts  we  considered  in  this  case 
study.  Moreover,  these  excluded  parts  seem  to  have  many  similarities  with  the  parts  we  considered  since 
they  essentially  contain  “rules”  for  other  programming  languages. 

The  Java  rules  in  PMD  contain  9  iterations  over  keys  or  entries  of  variable  declaration  maps  and  3  itera¬ 
tions  over  keys  or  entries  of  method  declaration  maps.  These  maps  come  from  “scope”  objects  that  represent 
essentially  type  bindings  in  PMD’s  AST.  We  annotated  the  respective  methods  to  return  permissions  for  the 
maps,  but  since  we  excluded  the  AST  and  binding  information,  i.e.,  the  scopes,  from  our  case  study,  it  is  not 
guaranteed  by  Plural  that  the  maps  arc  not  modified  while  rules  iterate  through  their  entries  or  keys. 

As  we  will  see  in  the  next  section,  maps  arc  not  considered  in  the  tracematch-based  analyses  that  we  aim 
to  compare  Plural  with  (although  4  of  the  iterations  in  question  nonetheless  apparently  confuse  Bodden’s 
analysis).  Therefore,  excluding  scopes  has  limited  impact  on  our  comparison.  Nonetheless,  we  discuss 
applying  Plural  to  them  below,  since  they  reveal  an  interesting  tooling  limitation. 

We  manually  inspected  the  code  and  determined  that  these  maps  arc  not  modified  when  rules  arc  exe¬ 
cuting.  This  is  because  PMD  follows  the  following  steps  in  processing  a  given  Java  source  file. 

1 .  Parsing.  Parsing  creates  AST  objects  and  fixes  their  references  to  their  respective  parents  and  children. 
The  parser  code  is  generated  using  the  parser  generator  jjTree. 

2.  Find  declarations.  A  AST  visitor  is  used  to  create  the  tree  of  scopes  and  find  all  declarations  in  a  given 
AST.  AST  node  fields  arc  set  to  point  to  their  scopes,  and  scopes  reference  their  parents.  Declarations 
arc  added  to  the  nearest  enclosing  scope  when  they  arc  encountered  by  the  visitor.  After  this  pass  over 
the  AST,  the  scope  tree  is  fixed. 

3.  Bind  name  occurrences  to  declarations.  A  second  AST  visitor  binds  names  in  the  AST  to  their 
declarations  and  modifies  the  declarations  to  reference  all  their  occurrences.  Only  after  this  pass, 
nodes  and  scopes  arc  fully  initialized  and  not  modified  subsequently. 

4.  Rule  checking.  Now  the  rules  are  run,  which  do  not  modify  AST  nodes  or  scopes  but  use  information 
from  scopes  about  name  occurrences  of  given  declarations  as  well  as  the  methods  declared  in  a  class, 
as  mentioned  above.  Rules  arc  also  typically  implemented  as  AST  visitors. 

Thus  PMD  clearly  follows  a  protocol  in  building  up  and  manipulating  AST  nodes  and  scopes  in  several 
phases,  before  running  rules  over  the  now  immutable  AST.  Notice  that  this  intuition  is  not  currently  enforced 
in  the  PMD  codebase:  nothing  prevents  rules  from  changing  or  deleting  the  binding  information  since  scopes 
are  defined  with  one  interface  Scope  that  allows  both  manipulating  binding  information  and  retrieving  the 
resulting  maps. 

Scopes  form  a  tree  with  parent  pointers  (but  without  child  pointers),  and  AST  nodes  likewise  form  a 
tree  with  parent  pointers.  Some,  but  not  all,  AST  nodes  have  scopes  associated  with  them,  and  the  two  tree 
structures  arc  parallel  to  each  other. 
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Tree  structures  are  a  well-known  challenge  for  program  verification  methodologies  that  currently  require 
manual  (but  sometimes  automatically  checkable)  proofs.8  We  did  develop  a  way  of  handling  tree  structures, 
such  as  the  tree  of  scopes  built  up  by  PMD  (Bierhoff  and  Aldrich,  2008),  but  Plural  is  not  equipped  to 
apply  this  solution  automatically.  The  idea  is  to  distribute  permissions  for  tree  nodes  amongst  its  parent  and 
children  and  assemble  them  when  nodes  need  to  be  modified.  Among  other  things,  this  requires  an  invariant 
that  the  permissions  in  a  list  of  children  combined  together  represent  a  full  permission  for  a  given  node, 
which  we  cannot  express  with  Plural’s  annotations.  This  also  requires  unpacking  multiple  objects  at  once, 
which  Plural  does  not  permit  (see  Section  6.3.7). 

We  have  at  least  two  other  choices  for  enforcing  that  scopes  arc  only  manipulated  before  rules  start 
iterating  the  bindings. 

•  We  could  define  a  typestate -based  protocol  for  scopes.  We  would  have  to  use  share  permissions  for 
referencing  scopes  (since  they  arc  aliased  through  all  their  children),  which  would  make  Plural  require 
dynamic  state  tests  when  calling  at  least  some  of  the  scope  methods. 

•  A  third  option  is  to  leverage  related  research  results.  In  particular,  Dean  Sutherland  has  successfully 
used  thread  colors  (Sutherland,  2008)  to  enforce  execution  phases.  In  his  work,  phases  arc  associated 
with  different  thread  colors,  and  the  scope  methods  would  require  the  respective  color.  (Notice  the 
similarity  to  the  previous  solution.)  It  appeal's  that  thread  coloring  would  not  be  able  to  enforce  that 
binding  maps  are  indeed  immutable  when  iterated.  This  leaves  the  author  wondering  whether  the  two 
approaches  could  be  beneficially  (and  soundly)  combined  to  leverage  their  strengths  and  compensate 
their  weaknesses. 

•  Finally,  one  could  think  about  think  about  special  support  for  traversing  trees.  For  instance,  when 
visiting  a  node,  one  could  (implicitly  or  explicitly)  have  permissions  for  all  parents  available.  (This 
is  essentially  the  solution  that  separation  logic  offers  (Jacobs  and  Piessens,  2008).)  This  also  leads 
to  the  observation  that  tree  traversals  only  seem  to  occur  in  certain  kinds  of  programs,  for  instance, 
compilers,  for  which  more  specialized  solutions  may  be  appropriate. 

In  summary,  the  PMD  scopes  are  hard  to  handle  and  may  in  fact  constitute  a  interesting  challenge 
problem  for  future  research  in  this  area. 

7.3.3  Comparison  to  Tracematches 

The  two  protocols  we  studied  were  also  checked  with  state-of-the-art  whole-program  protocol  checkers 
based  on  Tracematches  in  the  same  version  of  PMD  (Naeem  and  Lhotak,  2008;  Bodden  et  al.,  2008).  This 
section  uses  the  results  of  our  case  studies  with  PMD  for  comparison  with  these  analyses.  Unfortunately, 
only  Bodden  et  al.  (2008)  make  their  implementation  available  and  even  published  the  locations  of  false 
positives  produced  by  their  analysis;  therefore,  we  cannot  directly  compare  to  Naeem  and  Lhotak  (2008). 

Iterator  usage.  Precision  in  checking  the  iterator  protocol  is  comparable:  Plural  gives  3  false  positives; 
Tracematches  check  the  iterator  protocol  in  PMD  with  6  (Bodden  et  al.,  2008)  and  2  (Naeem  and  Lhotak, 
2008)  remaining  warnings. 

8The  2008  installation  of  the  International  Workshop  on  Specification  and  Verification  of  Component-Based  Systems 
(SAVCBS'08)  dedicated  a  challenge  problem  with  several  interesting  solutions  to  this  problem. 
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False  positives  reported  by  Bodden  et  al.  (2008) 
Ruled  out  with  Plural 
Conditionally  ruled  out  with  Plural 
Not  considered 


63 

44 

7 

12 


Table  7.4:  Plural's  performance  on  false  positives  in  checking  concurrent  modifications  reported  by  Bodden 
et  al.  (2008) 

The  pattern  that  causes  three  false  positives  when  using  Plural —  making  sure  a  set  is  non-empty  be¬ 
fore  retrieving  one  element  from  an  iterator — also  causes  false  positives  in  the  Tracematch-based  analyses. 
Naeem  and  Lhotak  (2008)  even  consider  them  true  positives  and  suggest  that  this  pattern  is  the  only  source 
of  warnings  flagged  by  their  analysis  in  PMD.  It  is  worth  pointing  out  that  both  Tracematch-based  analyses 
exclude  code  that  is  never  run  (according  to  the  call  graph)  from  consideration,  which  may  explain  why 
Naeem  and  Lhotak  (2008)  report  one  less  warning  (2)  than  Plural  (3). 

The  previously  mentioned  method  in  PMD  that  requires  its  iterator  argument  to  be  in  the  available 
state  accounts  for  the  remaining  warnings  reported  for  PMD  by  Bodden  et  al.  (2008);  these  warnings  arc 
considered  true  positives  by  Bodden  et  al.  (2008).  Thus,  our  approach  is  more  precise  compared  to  Bodden 
et  al.  (2008)  and  equally  precise  compared  to  Naeem  and  Lhotak  (2008)  on  the  simple  iterator  protocol.  We 
additionally  used  Plural  to  check  that  the  single  use  of  the  remove  method  in  PMD  respects  the  protocol 
(must  be  called  at  most  once  per  call  to  next),  which  was  not  considered  with  Tracematches. 

Concurrent  modification.  Our  approach  can  rule  out  concurrent  modifications  of  collections  and  maps 
where  the  Tracematch  analysis  of  Bodden  et  al.  (2008)  cannot.  Naeem  and  Lhotak  (2008)  and  Bodden  et  al. 
(2008)  report  very  similar  numbers  of  false  positives  for  this  protocol,  but  we  do  not  know  if  the  analysis 
proposed  by  Naeem  and  Lhotak  (2008)  sufferes  from  imprecisions  in  the  same  places  as  Bodden  et  al. 
(2008),  since  Naeem  and  Lhotak  (2008)  did  not  publish  an  implementation  of  their  analysis  or  the  locations 
where  false  positives. 

Both  Tracematch  analyses  report  a  large  number  of  false  positives  in  checking  concurrent  modifications 
in  PMD.  Naeem  and  Lhotak  (2008)  report  44  /  49  false  positives.  Bodden  et  al.  (2008)  report  63  /  100  false 
positives.9  51  out  of  their  63  false  positives  occur  in  parts  of  PMD  considered  by  our  case  study. 

Table  7.4  shows  that  Plural  could  rule  out  all  51  false  positives  reported  by  Bodden  et  al.  (2008)  in 
packages  we  considered.  Four  of  these  are  ruled  out  by  making  assumptions  about  scopes;  three  occur  in 
methods  where  Plural  reports  unrelated  false  positives. 

Plural  reports  false  positives  unrelated  to  the  ones  reported  with  Tracematches  for  three  reasons: 

•  Bodden  et  al.  (2008)  seem  to  better  support  language  features  that  are  challenging  for  Plural  (see 
discussion  of  ’’sources  of  imprecision”  in  Section  7.3.2),  including  reflection,  arrays,  and  static,  su¬ 
perclass,  and  foreign  fields. 

•  Tracematch  analyses  do  not  consider  collections  that  arc  never  iterated;  we  tracked  all  collections  in 
our  case  study. 

9Both  analyses  rule  out  unreachable  code,  but  we  do  not  know  why  the  number  of  potential  failure  points  differ. 
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•  The  protocols  checked  by  Naeem  and  Lhotak  (2008)  and  Bodden  et  al.  (2008)  only  check  concurrent 
modification  of  collections  while  their  entries  arc  iterated;  they  ignore  concurrent  modification  of 
maps  as  is  evident  from  the  Tracematch  protocols  published  for  both  Tracematch  papers  (Bodden, 
2009).  Our  case  study  considered  concurrent  modifications  of  both  maps  and  collections. 

A  follow-up  paper  (Bodden  et  al.,  2009)  checks  for  some  concurrent  modifications  of  maps,  but  not 
others  (Bodden,  2009).  The  later  paper  reports  precision  in  terms  of  runtime  instrumentation  code  that 
could  be  removed  because  the  absence  of  protocol  violations  could  be  proven  at  compile-time.  Only 
12%  each  of  instrumentation  points  for  the  patterns  related  to  concurrent  modifications  of  collections 
and  maps  could  be  removed.  The  remaining  instrumentation  causes  PMD  to  run  for  more  than  10 
hours,  compared  to  13  seconds  without  instrumentation  (Bodden  et  al.,  2009). 10 

The  bottom  line  is  that  using  Bodden  et  al.  (2008)  and  Plural  together  can  rule  out  all  concurrent  modi¬ 
fications  considered  by  both  analyses.  This  is  partially  due  to  their  different  strengths:  Bodden  et  al.  (2008) 
support  certain  Java  language  features  better  than  Plural  currently  does.  They  also  only  consider  executed 
code  and  collections  that  are  actually  iterated.  Plural,  on  the  other  hand,  seems  to  be  better  at  tracking  objects 
through  the  heap,  for  instance  through  fields  and  nested  collections,  which  is  crucial  in  checking  concurrent 
modifications  in  PMD.  We  suspect  that  comparison  with  Naeem  and  Lhotak  (2008)  yields  a  similar  result. 
This  suggests  that  our  approach  adds  precision  to  existing  whole-program  analyses  in  checking  challenging 
protocols. 


7.4  Discussion:  Lessons  Learned 

7.4.1  APIs 

Protocols  are  very  common.  Our  work  with  the  Java  standard  library  confirms  a  common  research  in¬ 
tuition:  protocols  are  very  common.  Considering  that  even  Java  exceptions  have  a  initialization  protocol 
(whose  violation  can  produce  an  exception!),  the  author  dares  to  say  that  protocols  arc  everywhere. 

Challenging  common  patterns.  There  were  at  least  three  common  challenges  that  we  found  across  sev¬ 
eral  of  the  APIs  we  specified. 

1.  We  were  surprised  how  prevalent  dynamic  state  test  methods  arc,  and  how  important  they  arc  in 
practice.  We  found  dynamic  state  test  methods  in  JDBC,  Collections,  and  regular  expressions,  and 
a  large  number  of  them  in  JDBC  alone.  For  example,  the  method  hasNext  in  the  Java  Iterator 
interface  tests  whether  another  element  is  available  (cf.  Section  3.1),  and  isEmpty  tests  whether  a 
collection  is  empty.  Since  such  tests  are  a  crucial  part  of  JDBC’s  facilities  for  executing  arbitrary  SQL 
commands,  it  was  crucial  for  handling  the  Beehive  code  that  our  approach  can  express  and  benefit 
from  the  tests. 

10  A  direct  comparison  with  these  numbers  is  difficult  because  we  do  not  know  exactly  how  instrumentation  points  are  counted.  As 
a  rough  approximation,  there  are  42  calls  to  Iterator  .next,  33  calls  to  Collection,  add,  and  3  calls  to  Map  .put  in  PMD  rules 
for  Java  (package  pmd. rules).  These  calls  should  represent  a  subset  of  the  instrumentation  points  used  for  checking  concurrent 
modifications  with  Tracematches.  Plural  reports  33  false  positives  in  this  part  of  the  codebase,  which  means  that  (at  least)  58%  of 
these  instrumentation  points  are  ruled  out  by  Plural  to  cause  concurrent  modifications  in  this  part  of  the  codebase. 
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2.  We  also  found  protocols  involving  multiple  interdependent  objects  in  these  APIs  (and  very  prevalent 
in  JDBC).  We  could  model  these  protocols  by  capturing  and  later  releasing  permissions. 

3.  We  used  method  cases  in  JDBC  and  the  Collections  API.  Method  cases  can  be  used  to  specify  full 
Java  iterators,  which  may  modify  the  underlying  collection  (Bierhoff,  2006). 

4.  We  also  used  marker  states  (also  known  as  type  qualifiers)  in  several  APIs,  and  in  particular  in  the 
Collections  API  (Section  7.1.2). 

We  believe  that  these  are  crucial  to  address  in  any  practical  protocol  checking  approach;  our  approach 
was  expressive  enough  to  handle  these  challenges  in  the  APIs  studied  in  this  section.  In  contrast,  previous 
approaches  rarely  or  incompletely  handle  these  patterns. 

Temporary  borrowing.  We  found  several  uses  of  aliasing  which  represent  variations  on  temporarily  “bor¬ 
rowing”  an  alias  (permission): 

1 .  Most  methods  borrow  permissions  for  their  parameters  and  return  the  permissions  to  the  caller  when 
they  return. 

2.  Some  API  objects  temporarily  “capture”  an  alias  which  they  only  release  when  they  are  no  longer  used 
or  explicitly  closed.  In  simple  cases,  such  as  regular-  expressions,  aliases  are  permanently  captured  in 
another  object. 

3.  Aliases  are  temporarily  borrowed  “from”  another  object.  This  is  in  particular-  one  way  of  looking  at 
many  of  the  “getters”  that  objects  define  for  giving  access  to  their  fields:  the  objects  temporarily  lend 
access  to  the  field  to  the  caller.  In  other  cases,  that  is  explicitly  not  the  intention,  and  instead,  getters 
return  a  copy  of  the  field  to  the  caller.  Permissions  can  cleanly  distinguish  between  the  two. 

7.4.2  API  Client  Code 

Aliasing  of  API  objects.  Beehive  excessively  aliases  objects  from  the  JDBC  API  in  fields  of  multiple 
objects,  and  PMD  frequently  aliases  Java  Collection  API  objects  from  a  field  in  views  or  iterators  on  the 
stack.  Aliasing  in  PMD  appears  to  be  relatively  controlled  (collections  are  for  example  not  referenced  in 
multiple  fields),  while  Beehive  uses  many  aliases.  Thus,  client  programs  seem  to  make  extensive  use  of 
aliasing. 

Internal  protocols  not  enforced.  We  saw  a  fair  amount  of  protocols  inside  the  client  codebases  we  stud¬ 
ied:  Beehive  expects  its  clients  to  follow  a  certain  protcol,  and  PMD  has  a  protocol  for  building  and  querying 
symbol  tables  (scopes).  The  similarities  between  these  are  striking: 

•  Neither  protocol  is  documented,  making  it  hard  for  Beehive  clients  or  new  PMD  developers  to  follow 
them. 

•  Neither  protocol  is  enforced  at  run-time;  thus,  if  the  protocol  is  violated,  it  will  lead  to  very  subtle 
bugs  in  the  program.  For  example,  changing  the  order  in  which  PMD  rules  are  run  on  a  compilation 
unit  could  change  the  violations  found. 
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•  Both  protocols  are  induced  by  protocols  in  the  underlying  APIs:  Beehive’s  protocol  comes  from 
the  need  to  close  result  sets  once  they  arc  no  longer  used,  and  PMD’s  protocol  avoids  concurrent 
modifications. 

Tree  structures.  Unsurprisingly,  tree  structures  arc  challenging  for  static  reasoning  about  programs,  if 
there  arc  parent  pointers  involved.  While  we  can  capture  these  structures  with  permissions  (Bierhoff  and 
Aldrich,  2008)  we  hope  that  future  work  can  simplify  working  with  trees  automatically. 

7.4.3  Plural  Tool  Usage 

Incremental  benefit.  Using  Plural  offers  incremental  benefit  in  three  ways: 

1.  Checking  protocols  for  different  APIs  is  largely  orthogonal,  allowing  the  incremental  checking  of 
more  and  more  protocols. 

2.  Shallow  protocols  involving  only  one  object  arc  easier  to  check  than  more  interesting  protocols,  as 
seen  in  the  two  parts  of  the  PMD  case  study.  This  allows  the  programmer  to  move  from  simple  to 
more  complex  checks. 

3.  Protocol  compliance  can  be  checked  in  parts  of  a  codebase,  with  annotations  creating  cutpoints  to 
other  parts.  In  PMD  we  successfully  isolated  Java  rules  from  AST  and  scopes,  at  the  price  of  making 
(manually  reviewable)  assumptions  about  how  the  excluded  parts  will  affect  the  analyzed  paid  of  the 
program. 

Iterative  process.  We  noticed  the  following  iterative  process  in  using  the  tool  on  existing  code:  we  first  ran 
Plural  “out  of  the  box”  on  a  codebase,  which  results  in  warnings  where  the  code  calls  into  the  APIs  whose 
protocols  the  tool  is  checking.  Adding  annotations  for  method  parameters  or  state  invariants  will  “move” 
these  errors  to  callers  of  the  methods  that  were  originally  causing  warnings.  Adding  annotations  to  the 
callers  will  move  the  warnings  again  until  we  reach  the  place  where  the  API  objects  that  these  annotations 
track  are  being  created.  Thus,  the  number  of  annotations  needed  directly  corresponds  to  how  far  in  the  call 
graph  a  API  object  “travels”  between  creation  and  use. 

Uniqueness  or  consistency.  In  PMD  we  discovered  an  interesting  trade-off  between  using  unique  permis¬ 
sions  and  giving  objects  a  simple  state  machine: 

•  We  tracked  “report”  objects  (which  collects  the  violations  found  by  PMD  rules)  using  unique  per¬ 
missions.  Consequently,  the  Report  class  only  needs  a  state  invariant  for  alive  (which,  therefore,  is 
comparable  to  a  traditional  class  invariant  in  JML  or  Spec#)  because  unpacking  a  unique  permission 
saves  us  from  having  to  pack  to  an  intermediate  state  before  every  method  call. 

•  Conversely,  we  tracked  rules  using  full  permissions,  which  made  it  necessary  to  define  a  quiet  state  for 
rules.  Invariants  for  rule  fields  are  then  associated  with  the  quiet  state.  When  rule  implementations  call 
methods  they  have  to  pack  to  some  other  state.  Thus,  the  added  aliasing  flexibility  of  full  permissions 
comes  at  the  price  of  needed  states  beyond  alive  for  specifying  and  verifying  rules. 
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Checked  Annotation  rate  False  positive  rate  Plural  runtime 

4  APIs  in  Beehive  1  per  30  lines  1  per  400  lines  188ms  per  method 

Collections  in  parts  of  PMD  1  per  25  LOC  1  per  188  LOC  1 19ms  per  method 

Iterator  protocol  in  all  of  PMD  1  per  2,600  LOC  1  per  13,000  LOC  62ms  per  method 


Table  7.5:  Overhead  and  precision  in  Beehive  and  PMD  case  studies 


This  is  consistent  with  our  experience  using  Spec#  (Barnett  et  ah,  2004),  whose  ownership  model  has 
similarities  to  full  permissions  and  guarantees  class  invariants  to  hold  only  when  objects  are  in  a  state 
called  “consistent”. 

Annotations  like  types.  Throughout  our  case  studies  we  used  one  or  less  than  one  annotation  per  method. 
This  number  would  of  course  go  up  if  every  object  in  the  program  had  protocols  we  wanted  to  track.  (Plural 
ignores  objects  that  have  no  protocol.)  The  annotations  we  provided,  however,  are  in  their  extent  very 
comparable  to  conventional  Java  typing  information  for  fields  and  method  parameters.  This  suggests  that 
the  overhead  of  tracking  permissions  in  a  program  is  on  par  with  what  Java  developers  do  today  to  make 
their  program  compile  with  the  Java  compiler. 

Promising  precision.  Plural’s  false  positive  rates  (Table  7.5)  represent  a  trade-off  with  the  overhead  of 
providing  annotations.  Removing  common  sources  of  false  positives,  such  as  inheritance  and  static  fields, 
would  substantially  improve  the  false  positive  rates  to  around  1  per  at  least  1 ,000  lines  of  code  in  all  case 
studies,  which  we  believe  to  be  fairly  impressive  considering  the  challenging  protocols  checked. 

Good-enough  performance.  Plural’s  performance  turned  out  to  be  “good  enough”  to  be  used  during 
development.  Individual  methods  could  be  checked  on  average  in  around  100ms.11  Delays  occur  in  practice 
when  “choice  contexts”  multiply  with  frequent  packing  and  unpacking  (because  Plural  tries  all  possible 
states  to  pack  to)  and  extensive  use  of  method  cases  (because  Plural  tries  all  cases).  In  many  cases,  however, 
choices  are  limited  quickly  (e.g.,  due  to  unsatisfiable  state  invariants),  which  limits  longer  analysis  times  to 
a  few  cases. 


11  This  estimate  includes  measurements  on  Nels  Beckman's  multi-threaded  benchmark  programs.  Performance  was  measured  on 
a  Dell  PC  running  Windows  XP  Service  Pack  2,  with  a  3.2GHz  Intel  Pentium  4  and  2GB  of  RAM.  Thanks  to  Nels  Beckman  for 
collecting  performance  information. 


Chapter  8 

Related  Work 


This  section  compares  this  research  to  related  work.  A  variety  of  different  protocol  specification  approaches 
has  been  proposed  in  the  literature.  Such  specifications  arc  primarily  used  for  three  different  purposes,  which 
we  discuss  in  this  order:  checking  programs  for  protocol  compliance,  protocol  checking  or  simulation  at 
runtime,  and  checking  component  compositionality  at  the  design  level.  Afterwards,  we  discuss  fractional 
permission  inference,  comprehensive  program  verification  approaches,  and  protocol  inference. 


8.1  Static  Protocol  Analyses 

This  section  focuses  on  static  analyses  for  checking  protocol  compliance.  We  first  discuss  modular  and  then 
whole -program  protocol  checking  approaches. 

A  plethora  of  approaches  was  proposed  in  the  literature  for  checking  protocol  compliance.  These  ap¬ 
proaches  differ  significantly  in  the  way  protocols  arc  specified,  including  typestates  (Strom  and  Yemini, 
1986;  DeLine  and  Fahndrich,  2001,  2004b;  Kuncak  et  al.,  2006;  Fink  et  ah,  2006;  Bierhoff  and  Aldrich, 
2007b),  type  qualifiers  (Foster  et  ah,  2002;  Aiken  et  ah,  2003;  Chin  et  ah,  2005a),  size  properties  (Chin 
et  ah,  2005b),  direct  constraints  on  ordering  (Igarashi  and  Kobayashi,  2002;  Tan  et  ah,  2003),  tracematches 
(Bodden  et  ah,  2008;  Naeem  and  Lhotak,  2008),  type  refinements  (Mandelbaum  et  ah,  2003;  Degen  et  ah, 
2007),  “predicates”  (Perry,  1989),  and  various  temporal  logics  (e.g.  Henzinger  et  ah,  2002,  and  others  dis¬ 
cussed  below).  In  these  approaches,  as  in  ours,  usage  rules  of  the  API(s)  of  interest  have  to  be  codified  by  a 
developer. 

Once  usage  protocols  arc  codified,  violations  can  be  detected  statically  (like  in  our  and  most  of  the  above 
approaches)  or  dynamically  while  the  program  is  executing  (e.g.  Bierhoff  and  Aldrich,  2005;  Dwyer  et  ah, 
2007).  Many  static  approaches  arc  modular  like  ours  but  there  are  also  whole -pro gram  analyses  that  require 
no  or  minimal  developer  intervention  (e.g.  Foster  et  ah,  2002;  Henzinger  et  ah,  2002).  Unlike  previous 
modular  approaches,  our  approach  does  not  require  precise  tracking  of  all  object  aliases  (e.g.  DeLine  and 
Fahndrich,  2001,  2004b)  or  impose  an  ownership  discipline  on  the  heap  (e.g.  Barnett  et  ah,  2004))  in  order 
to  be  modular.  This  section  discusses  modular  and  global  static  protocol  analyses;  dynamic  analyses  arc 
discussed  in  the  next  section. 

Our  approach  complements  research  on  helping  developers  accomplish  their  goals  with  the  “right” 
framework  interactions,  e.g.,  using  “design  fragments”  (Fairbanks,  2007):  our  approach  allows  checking 
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whether  the  concrete  combination  of  framework  interactions  chosen  by  a  developer  is  permitted  by  the 
framework. 

8.1.1  Modular  Analyses 

Many  of  the  proposed  static  approaches,  including  ours,  arc  modular  and  require  developer-provided  an¬ 
notations  in  the  analyzed  code  in  addition  to  codifying  API  usage  rules.  Only  a  few  of  these  approaches 
(DeLine  and  Fahndrich,  2004b;  Tan  et  ah,  2003;  Degen  et  al.,  2007)  target  object-oriented  languages  like 
our  approach. 

Our  approach  provides  a  great  deal  of  flexibility  in  aliasing  objects,  which  will  be  discussed  in  more 
detail  below.  Furthermore,  none  of  the  existing  modular  approaches  includes  the  concept  of  temporary  state 
information  (knowledge  about  the  current  state  of  an  object  that  has  to  be  “forgotten”  due  to  possible  effects 
through  aliases). 

Checking  protocol  implementation  code.  Ours  is  one  of  the  few  approaches  that  can  reason  about  cor¬ 
rectly  implementing  APIs  independent  from  their  clients.  (Interestingly,  all  of  these  approaches  that  we  arc 
aware  of  arc  modular  typestate  analyses:  DeLine  and  Fahndrich  (2004b);  Kuncak  et  al.  (2006);  Bierhoff 
and  Aldrich  (2007b).)  Ours  is  the  only  approach  (that  we  are  aware  of)  that  can  verify  correct  usage  and 
implementation  of  dynamic  state  test  methods.  Several  other  approaches  can  verify  their  correct  usage  (e.g. 
Mandelbaum  et  al.,  2003;  Chin  et  al.,  2005b),  but  not  their  implementation. 

We  extended  several  ideas  for  checking  API  implementation  code  from  Fugue  (DeLine  and  Fahndrich, 
2004b)  to  work  with  access  permissions  including  state  invariants,  packing,  and  frames.  In  contrast  to  Fugue, 
subtypes  are  always  fully  substitutable  with  our  approach  and  arc  allowed  to  change  inherited  protocols,  and 
subclasses  arc  free  to  use  or  not  use  code  inherited  from  superclasses. 

Dealing  with  aliasing.  A  simple  mechanism  to  preserve  soundness  in  modular  protocol  analyses  is  to 
prevent  aliasing  by  only  allowing  linear  references  to  stateful  objects  (Wadler,  1990).  But  programming  with 
lineal-  types  is  very  inconvenient  because  whenever  an  object  is  passed  to  a  function  (or  used  otherwise),  it 
disappears  at  the  call  site.  Therefore,  a  variety  of  mechanisms  to  relax  linearity  were  proposed.  Uniqueness, 
sharing,  and  immutability  (Aldrich  et  al.,  2002;  Boyland  and  Retert,  2005)  have  been  put  to  use  in  resource 
usage  analysis  (Igarashi  and  Kobayashi,  2002;  Chin  et  al.,  2005b).  Alias  types  (Crary  et  al.,  1999;  Smith 
et  al.,  2000)  allow  multiple  variables  to  refer  to  the  same  object  but  require  a  linear  token  for  object  accesses 
that  can  be  borrowed  (Boyland  and  Retert,  2005)  during  function  calls.  Focusing  can  be  used  for  temporary 
state  changes  of  shared  objects  (Fahndrich  and  DeLine,  2002;  Foster  et  al.,  2002;  Barnett  et  al.,  2004; 
Morrisett  et  al.,  2005).  Adoption  prevents  sharing  from  leaking  through  entire  object  graphs  (as  in  DeLine 
and  Fahndrich,  2004b)  and  allows  temporary  sharing  until  a  linear  adopter  is  deallocated  (Fahndrich  and 
DeLine,  2002).  All  these  techniques  need  to  be  aware  of  all  references  to  an  object  in  order  to  change  its  (or 
its  adoptees’)  state. 

Access  permissions  allow  state  changes  even  if  objects  are  aliased  from  unknown  places.  Moreover, 
access  permissions  give  fine-grained  access  to  individual  data  groups  (Leino,  1998).  States  and  fractions 
(Boyland,  2003)  let  us  capture  alias  types,  borrowing,  adoption,  and  focus  with  a  single  mechanism.  Sharing 
of  individual  data  groups  has  been  proposed  before  (Boyland  and  Retert,  2005),  but  it  has  not  been  exploited 
for  reasoning  about  object  behavior.  In  Boyland’s  work  (Boyland,  2003),  a  fractional  permission  means 


8. 1 .  STATIC  PROTOCOL  ANALYSES 


115 


immutability  (instead  of  sharing)  in  order  to  ensure  non-interference  of  permissions.  We  use  permissions 
to  keep  state  assumptions  consistent  but  track,  split,  and  join  permissions  in  the  same  way  as  Boyland. 
Concurrently  to  us,  Java(X)  (Degen  et  al.,  2007)  proposed  “activity  annotations”  that  can  be  seen  as  full, 
share,  and  pure  permissions  for  whole  objects  that  can  be  split  but  not  joined. 


Empirical  results.  Previous  modular  approaches  arc  often  proven  sound  and  shown  to  work  for  well- 
known  examples  such  as  file  access  protocols.  But  automated  checkers  arc  rare,  and  case  studies  with  real 
APIs  and  third-party  code  hard  to  find.  Notable  exceptions  include  Vault  (Define  and  Fahndrich,  2001)  and 
Fugue  (Define  and  Fahndrich,  2004b,a),  which  arc  working  automated  checkers  that  were  used  to  check 
compliance  to  Windows  kernel  and  .NET  standard  library  protocols,  respectively  (although  Vault  requires 
rewriting  the  code  into  its  own  C-like  language). 

This  dissertation  shows  that  our  approach  can  be  used  in  practical  development  tools  for  enforcing  real 
API  protocols.  As  far  as  we  know,  this  is  the  first  work  that  reports  on  challenges  and  recurring  patterns 
in  specifying  typestate  protocols  of  large,  real  APIs.  We  also  report  overhead  (in  terms  of  annotations)  and 
precision  (in  terms  of  false  positives)  in  checking  open-source  codebases  with  our  tool. 

We  suspect  that  empirical  results  arc  sparse  because  APIs  such  as  the  ones  discussed  in  Chapter  7 
would  be  difficult  to  handle  with  existing  modular  approaches  due  to  their  limitations  in  reasoning  about 
aliased  objects.  These  limitations  make  it  difficult  to  specify  the  object  dependencies  we  found  in  the  JDBC, 
Collections,  and  Regular  Expressions  APIs  in  the  Java  standard  library.  Fugue,  for  instance,  was  used  for 
checking  compliance  with  the  .NET  equivalent  of  JDBC,  but  the  published  specification  does  not  seem  to 
enforce  that  connections  remain  open  while  “commands”  (the  .NET  equivalent  of  JDBC  “statements”)  arc 
in  use  (DeLine  and  Fahndrich,  2004a). 

Existing  work  on  permissions  recognized  these  challenges  (Boyland,  2003;  Boyland  and  Retert,  2005) 
but  only  supports  unique  and  immutable  permissions  directly  and  does  not  track  behavioral  properties  (such 
as  typestates)  with  permissions.  Similar  to  Fugue  (DeLine  and  Fahndrich,  2004b),  Java(X)’s  “activity  anno¬ 
tations”  do  not  allow  recovering  “exclusive”  access  to  an  object  once  it  becomes  shared  (Degen  et  al.,  2007), 
which  seems  to  prevent  their  use  for  encoding  the  iterator  and  JDBC  protocols. 

8.1.2  Whole-Program  Analyses 

Global  approaches  are  very  flexible  in  handling  aliasing  and  require  no  or  minimal  developer  intervention. 
Approaches  based  on  abstract  interpretation  (e.g.  Ball  and  Rajamani,  2001;  Das  et  al.,  2002;  Hallem  et  al., 
2002;  Fink  et  al.,  2006;  Naeem  and  Lhotak,  2008;  Bodden  et  al.,  2008)  typically  verify  client  conformance 
while  the  protocol  implementation  is  assumed  correct.  Sound  approaches  rely  on  a  global  aliasing  analysis 
(Ball  and  Rajamani,  2001 ;  Das  et  al.,  2002;  Fink  et  al.,  2006;  Naeem  and  Lhotak,  2008;  Bodden  et  al.,  2008). 
Likewise,  most  model  checkers  operate  globally  (e.g.  Henzinger  et  al.,  2002)  or  use  assume-guarantee  rea¬ 
soning  between  coarse-grained  static  components  (Giannakopoulou  et  al.,  2004;  Hughes  and  Bultan,  2007). 
For  instance,  the  Magic  tool  checks  individual  C  functions  but  has  to  inline  user-provided  state  machine 
abstractions  for  library  code  in  order  to  accommodate  aliasing  (Chaki  et  al.,  2003).  The  above  analyses 
typically  run  on  the  complete  codebase  once  a  system  is  fully  implemented  and  can  be  very  expensive.  Our 
approach  supports  developers  by  checking  the  code  at  hand  like  a  typechecker.  Thus  the  benefits  of  our 
approach  differ  significantly  from  global  analyses. 

In  contrast  to  modular  checkers,  many  whole -program  analyses  were  implemented  and  empirically  eval- 
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uated.  While  model  checkers  (Henzinger  et  ah,  2002)  typically  have  severe  limitations  in  scaling  to  larger 
programs,  approaches  based  on  abstract  interpretations  were  shown  to  scale  quite  well  in  practice.  “Sound” 
(see  below)  approaches  rely  on  a  global  aliasing  analysis  (Ball  and  Rajamani,  2001;  Das  et  ah,  2002;  Fink 
et  ah,  2006;  Bodden  et  ah,  2008;  Naeem  and  Lhotak,  2008)  and  become  imprecise  when  alias  information 
becomes  imprecise. 

Section  7.3.1  compares  our  approach  to  state-of-the-art  whole-program  analyses  based  on  tracematches 
(Bodden  et  ah,  2008;  Naeem  and  Lhotak,  2008)  using  one  of  their  benchmark  programs  (PMD).  The  com¬ 
parison  shows  that  our  approach  matches  the  precision  of  tracematch-based  analyses  for  checking  a  simple 
iterator  usage  protocol  with  extremely  low  developer  overhead.  Our  approach  could  additionally  rule  out 
most  false  positives  reported  by  Bodden  et  ah  (2008)  in  PMD  when  checking  the  more  challenging  concur¬ 
rent  modification  of  collections.  Another  previous  global  typestate  analysis  has  also  been  used  to  check  the 
simpler  iterator  usage  protocol,  but  in  a  different  corpus  of  client  programs  and  with  varying  precision  (Fink 
et  ah,  2006). 

These  whole-program  analyses  have  been  used  to  make  sure  that  dynamic  state  test  methods  are  called, 
but  not  that  the  test  actually  indicated  the  needed  state  (Fink  et  ah,  2006;  Bodden  et  ah,  2008;  Naeem  and 
Lhotak,  2008).  For  example,  the  protocols  they  checked  require  calling  hasNext  before  calling  next  in 
iterators,  but  they  do  not  check  whether  hasNext  returned  true,  which  with  our  approach  is  expressed 
and  ensured  easily.  Tracematch-based  analyses  (Bodden  et  ah,  2008;  Naeem  and  Lhotak,  2008)  currently 
lack  the  expressiveness  to  capture  these  protocols  more  precisely,  while  approaches  based  on  must-alias 
information  (e.g.  Fink  et  ah,  2006)  should  be  able  to,  but  do  not  in  their  published  case  studies,  encode 
these  protocols.  ESP  (Das  et  ah,  2002)  performs  an  analysis  that  can  distinguish  program  branches,  but  at 
least  the  published  example  does  not  suggest  that  ESP  can  capture  dynamic  state  tests.  This  is  arguably  an 
omission  in  these  approaches  that,  given  the  importance  of  dynamic  state  tests  in  practice,  we  believe  should 
be  addressed. 

The  Metal  tool  (Hallem  et  ah,  2002)  found  hundreds  of  protocol  violations  in  Linux  kernel  code.  Dy- 
naMine  identified  bug  patterns  related  to  protocol  violations  in  Eclipse  and  JEdit  using  statistical  methods 
(Livshits  and  Zimmermann,  2005).  These  approaches  are  unsound  in  general.  Our  approach  verifies  com¬ 
pliance  to  protocols  and  can  therefore  help  prevent  the  defects  found  in  these  studies. 

Cqual  is  a  whole-program  type  qualifier  inference  system  for  C  that,  like  Metal,  found  locking  protocol 
violations  in  the  Linux  kernel  (Foster  et  ah,  2002).  In  many  cases,  the  placement  of  “focus”  (or  “restrict”)  to 
temporarily  change  the  state  of  an  object  can  also  be  inferred  (Aiken  et  ah,  2003).  While  Cqual  allows  type 
qualifiers  to  change  over  time  (because  it  is  flow-sensitive)  it  that  type  qualifiers  cannot  encode  protocols 
involving  multiple  objects,  such  as  the  ones  found  in  JDBC. 

We  do  not  claim  to  comprehensively  compare  our  approach’s  precision  relative  to  whole-program  anal¬ 
yses.  But  we  do  point  out  that  our  approach,  unlike  whole-program  analyses,  can  reason  about  API  im¬ 
plementations  separately  from  clients  and  handles  dynamic  state  tests  soundly,  as  discussed  above.  Rea¬ 
soning  about  API  implementations  separately  from  clients  is  critical  for  libraries  such  as  Beehive  that  may 
have  many  clients.  Our  approach  also  seems  to  be  superior  to  state-of-the-art  tracematch-based  analyses  in 
checking  “deep”  protocols  in  challenging  codebases. 


8.2.  PROTOCOLS  AT  RUNTIME 


117 


8.2  Protocols  at  Runtime 

Protocols  can  also  be  used  to  check  or  simulate  running  systems.  Statecharts  (Harel,  1987)  arc  used  to 
visually  specify  reactive  systems  with  hierarchical  state  machines.  Statecharts  can  be  used  to  generate  exe¬ 
cutable  simulations  of  the  specified  system  (Hard  and  Naamad,  1996).  Butkevich  et  al.  (2000)  use  regular 
expressions  to  describe  protocols  (that  are  translated  into  labeled  transition  systems).  Runtime  checks  arc 
used  to  ensure  protocol  compliance  of  a  running  system.  We  previously  implemented  a  runtime  protocol 
checker  for  typestate  protocols  specified  with  union  and  intersection  types  (Bierhoff  and  Aldrich,  2005). 
Dwyer  et  al.  (2007)  shows  how  to  reduce  the  overhead  of  such  checks  and  recent  tracematch-based  analyses 
have  been  used  to  remove  runtime  checks  where  possible  (Bodden  et  al.,  2008,  2009). 

This  dissertation  proposes  a  static  protocol  verification  approach  that  does  not  require  a  runtime  pro¬ 
tocol  checker.  No  explicit  runtime  representation  of  access  permissions  is  needed.  However,  in  order  for 
verification  to  succeed,  runtime  tests  in  the  form  of  if  statements  may  be  required.  We  employ  mecha¬ 
nisms  similar  to  Statecharts'  AND-  and  OR-states  to  define  state  spaces.  Our  method  specifications  involve 
multiple  objects  and  are  therefore  quite  different  from  state  transitions  in  Statecharts. 

8.3  Verifying  Component  Compositions 

This  section  summarizes  approaches  for  checking  whether  protocols  of  independently  developed  compo¬ 
nents  are  compatible.  The  architecture  description  language  Wright  (Allen  and  Garlan,  1997)  includes 
protocol  specifications  based  on  CSP  (Hoare,  1985).  A  model  checker  for  CSP  is  used  to  verify  composi- 
tionality,  i.e.,  verify  that  components  arc  connected  in  such  a  way  that  they  do  not  deadlock.  FSP  (Magee 
et  al.,  1999)  specifies  component  behavior  with  labeled  transition  systems  (LTSs),  and  compositions  of  LTSs 
can  be  checked  for  compositionality  as  well.  This  check  adds  flexibility  to  assume-guarantee  reasoning  (Gi- 
annakopoulou  et  al.,  2004).  Interface  automata  (de  Alfaro  and  Henzinger,  2001)  describe  protocols  as  state 
machines  that  can  likewise  be  checked  for  compositionality. 

These  approaches  provide  design-level  reasoning  about  component  compositions.  Our  verification  ap¬ 
proach  essentially  checks  compositionality  as  well,  but  it  does  so  by  checking  the  actual  code  that  uses  an 
API  for  protocol  compliance. 

8.4  Fractional  Permission  Inference 

Fractional  permissions  were  proposed  by  Boyland  for  avoiding  data  races  in  concurrent  programs  (Boyland, 
2003)  and  have  since  been  used  in  a  variety  of  contexts.  In  particular,  we  introduce  new  kinds  of  permissions 
and  the  notion  of  state  guarantees;  we  use  linear  logic  to  compose  predicates  from  permissions;  and  we  track 
typestates,  which  Boyland  does  not  consider. 

This  dissertation  provides  a  permission  inference  system  for  a  fragment  of  linear  logic  predicates  that  is 
polymorphic  with  respect  to  fractions  and  describes  its  implementation  in  a  protocol  checking  tool  for  Java 
that  can  fully  check  individual  methods  in  100ms  on  average. 

Terauchi  and  Aiken  (2008)  previously  proposed  a  fraction  inference  system  for  checking  data  races  in 
concurrent  programs  whose  implementation  is  based  on  a  linear  programming  engine  (Terauchi,  2008).  In 
their  work,  control  flow  merges  introduce  new  fraction  variables  that  are  constrained  to  be  smaller  than  the 
incoming  fractions.  Function  signatures  arc  also  inferred  with  fraction  variables.  Constraints  can  then  be 
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collected  by  scanning  the  entire  program  once,  even  in  the  presence  of  conditionals  and  loops,  allowing 
constraint  collection  to  be  linear  in  the  size  of  the  program.  They  also  used  fractions  to  infer  legal  orderings 
of  side-effecting  operations  based  on  “witnesses”  (Terauchi  and  Aiken,  2005). 

Conversely,  we  collect  constraints  with  a  intra-procedural  flow  analysis,  which  is  polynomial  in  the 
worst  case  (Nielson  et  al.,  1999,  Chapter  6)  and  requires  annotations  for  permissions  passed  into  and  out  of 
methods.  An  advantage  of  our  modular  approach  is  that  permissions  can  be  consumed  in  loops,  which  was 
not  previously  possible  (see  Terauchi,  2008,  Section  5).  We  arc  also  able  to  use  universal  quantification  in 
function  signatures  (currently  provided  with  explicit  annotations),  which  may  be  more  flexible  than  inferring 
concrete  fractions  for  signatures.  These  differences  arc  due  to  our  support  for  polymorphic  fractions,  which 
arc  not  available  in  previous  work. 

Ongoing  work  in  Boyland’s  group  on  avoiding  data  races  unfortunately  elides  the  details  of  their  fraction 
inference  implementation  (Zhao,  2007).  Bornat  et  al.  (2005)  combined  Boyland’s  fractional  permissions 
with  separation  logic,  but  their  approach  is  intended  for  hand-written  proofs  about  program  correctness. 
VeriFast  is  an  automated  proof  checker  for  separation  logic  (Jacobs  and  Piessens,  2008)  that  to  our  knowl¬ 
edge  supports  Boyland’s  fractional  permissions,  but  the  details  of  the  implementation  arc  again  unknown. 
VeriFast  seems  to  require  significant  developer  input. 

Other  ongoing  work  encodes  Boyland’s  fractions  using  integers  representing  percentages  between  0  and 
100  as  well  as  “infinitesimal”  fractions  in  first-order  logic  (Leino  and  Muller,  2009).  Loop  invariants  have  to 
be  provided  by  hand,  while  our  implementation  infers  loop  invariants.  Unlike  with  polymorphic  fractions, 
it  appears  that  this  unusual  encoding  will  require  developers  to  use  concrete  percentage  values  or  concrete 
numbers  of  infinitesimal  fractions  in  most  of  their  method  pre-conditions  and  loop  invariants.  Infinitesimal 
fractions  come  in  a  duplicable  variety,  but  those  cannot  be  used  to  borrow  or  otherwise  temporarily  alias 
objects,  such  as  database  connections. 

8.5  Comprehensive  Program  Verification 

Comprehensive  program  verification  approaches  can  also  be  used  to  reason  about  protocols.  These  ap¬ 
proaches  employ  logical  predicates  in  specifications.  We  first  discuss  approaches  based  on  classical  first- 
order  logic  and  then  consider  separation  logic. 

First-order  logic.  Early  approaches  to  comprehensive  program  verification  include  Alphard  (Wulf  et  al., 
1976)  and  later  Larch  (Guttag  et  al.,  1985).  In  both  approaches,  datatypes  arc  specified  with  invariant  pred¬ 
icates,  and  their  operations  arc  specified  with  pre-  and  post-condition  predicates.  Alphard  uses  uniqueness 
for  controlling  aliasing  and  uses  “states”  with  invariants  for  more  concise  description  of  datatype  implemen¬ 
tations.  Eiffel  (Meyer,  1992)  pioneered  the  idea  of  executable  pre-  and  post-conditions.  Methods  in  Eiffel 
declare  their  pre-  and  post-conditions  using  Eiffel  code  and  check  them  dynamically. 

The  JML  is  a  behavioral  specification  language  for  Java  that  incorporates  ideas  such  as  datatype  invari¬ 
ants  from  Larch  and  executable  specifications  from  Eiffel  (Leavens  et  al.,  1999).  The  JML  (Leavens  et  al., 
1999)  is  very  rich  and  complex  in  its  specification  features;  it  is  more  capable  than  our  system  to  express 
object  behavior  (not  just  protocols),  but  also  potentially  more  difficult  to  use  due  to  its  complexity.  Veri¬ 
fying  JML  specifications  is  undecidable  in  the  general  case.  Tools  like  ESC/Java  (Flanagan  et  al.,  2002) 
can  partially  check  JML  specifications  but  arc  unsound  because  they  do  not  have  a  sound  methodology  for 
handling  aliasing.  Spec#  is  comparable  in  its  complexity  to  the  JML  and  imposes  similar  overhead.  The 
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Boogie  methodology  allows  sound  verification  of  Spec#  specifications  but  requires  programs  to  follow  an 
ownership  discipline  (Barnett  et  ah,  2004). 

Our  system  is  much  simpler  than  these  approaches,  focusing  as  it  does  on  protocols,  and  can  be  auto¬ 
mated  to  a  higher  degree  as  a  result.  Our  treatment  of  aliasing  makes  our  system  sound,  where  ESC/Java 
is  not.  While  the  treatment  of  aliasing  in  our  system  does  involve  complexity,  it  gives  the  programmer 
more  flexibility  than  Boogie’s  ownership  methodology  while  remaining  modular  and  sound.  Because  it  is 
designed  for  protocol  verification  in  particular,  our  system  will  generally  impose  smaller  specification  over¬ 
head  than  the  JML  or  Spec#.  Additionally,  ownership-based  approaches  cannot  express  dependencies  of 
multiple  objects  on  a  single  “server”  object,  while  permissions  allowed  us  to  express  these  patterns  for  three 
major  Java  APIs. 

Crucial  for  reasoning  about  object-oriented  programs  is  the  treatment  of  subtyping  and  inheritance.  Be¬ 
havioral  subtyping  formalizes  rules  for  checking  whether  subtypes  conform  to  the  behavior  of  supertypes 
(Liskov  and  Wing,  1994).  The  JML  and  Spec#  guarantee  behavioral  subtyping  through  specification  in¬ 
tersection  (Dhara  and  Leavens,  1996).  The  JML  has  no  particular  methodology  for  handling  inheritance. 
Spec#'s  handling  of  inheritance  is  based  on  Lugue’s  frames  (DeLine  and  Lahndrich,  2004b).  Our  approach 
includes  a  logic -based  check  for  behavioral  subtyping  of  protocol  specifications.  Like  Lugue  and  Spec#,  we 
handle  inheritance  using  frames.  Our  approach  of  unpacking  one  frame  at  a  time  differs  from  Lugue  but 
is  similar  to  Spec#’s  “local  expose”.  Unlike  in  Lugue,  the  states  of  different  frames  can  be  decoupled,  and 
method  overriding  requirements  arc  relaxed  in  our  approach. 

Separation  logic.  Separation  logic  extends  classical  logic  to  simplify  reasoning  about  shared  mutable 
state  (Reynolds,  2002).  Its  operators  arc  very  similar  to  those  defined  in  the  MALL  fragment  of  linear 
logic  (Girard,  1987)  and  can  be  combined  with  traditional  classical  logic  operators.  Separation  logic  is 
undecidable  in  the  general  case,  but  heap  shapes  have  been  successfully  inferred  automatically  based  on 
separation  logic  (Magill  et  ah,  2006;  Calcagno  et  al.,  2009). 

Separation  logic  completely  isolates  effects  on  separate  pa  its  of  the  heap  as  unique  permissions  do. 
Other  permissions  are  not  available  directly  in  separation  logic,  but  separation  logic  predicates  can  be  used 
to  pass  shared  objects  around  between  different  parts  of  the  program.  Immutability  can  be  natively  included 
by  combining  separation  logic  with  fractions  (Bornat  et  ah,  2005).  Parkinson  and  Bierman  (2008)  proposed 
“predicate  families”  for  using  separation  logic  in  the  context  of  object-oriented  programming.  Predicate 
families  have  many  similarities  to  our  state  invariants  and  consequently  provide  comparable  flexibility  to 
subclasses  in  using  inherited  code. 

8.6  Protocol  Inference 

Recent  static  API  protocol  inference  systems  for  C  (Henzinger  et  ah,  2005)  and  Java  (Nanda  et  ah,  2005) 
use  typestates  as  their  protocol  abstraction.  Remarkably,  the  inferred  Java  protocols  (Nanda  et  ah,  2005) 
are  very  similar  to  what  we  can  enforce.  Protocols  can  also  be  “mined”  from  revision  histories  (Livshits 
and  Zimmermann,  2005),  typically  by  employing  statistical  methods  to  identify  “common  patterns”.  These 
approaches  arc  complimentary  to  ours  in  that  the  inferred  protocols  could  be  specified  and  checked  using 
Plural,  which  could  reduce  the  annotation  burden  of  specifying  API  protocols. 
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Chapter  9 

Conclusions 


9.1  Validation  of  Hypotheses 

This  dissertation  set  out  to  demonstrate  that  typestate -based  protocols  with  access  permissions  can  enforce 
API  protocols  in  practical  object-oriented  software.  This  thesis  is  not  directly  testable;  therefore,  we  have 
focused  on  four  testable  hypotheses  that  aim  at  establishing  our  approach’s  well-foundedness  and  its  ap¬ 
plicability  to  common  software  practices,  in  particular  with  respect  to  APIs,  conventional  programming 
languages,  and  how  code  is  written  in  practice. 

Our  first  hypothesis  is  that  our  approach  can  succinctly  capture  common  protocols;  our  second  hypoth¬ 
esis  is  that  the  approach  can  soundly  and  modularly  check  protocols  in  object-oriented  code.  Our  third 
hypothesis  is  that  the  approach  can  be  sufficiently  automated  for  use  by  software  developers,  and  our  fourth 
hypothesis  is  that  our  approach  can  be  used  to  check  off-the-shelf  software. 

Notice  that  our  thesis  does  not  claim  that  our  approach  works  well  with  all  APIs  and  all  code  written  to 
date.  Instead,  we  have  attempted  to  “simulate”  our  approach’s  use  in  practice  by  considering  a  handful  of 
real,  commonly  used  APIs  as  well  as  third-party  open-source  software. 

The  following  sections  discuss  the  evidence  we  gathered  in  support  of  each  of  our  hypotheses. 

9.1.1  Capture  Common  Protocols  Succinctly 

Our  first  hypothesis  establishes  the  applicability  of  our  approach  for  API  protocols  occurring  in  practice. 

Typestate -based  specifications  with  access  permissions  can  succinctly  capture  API  protocols 
commonly  occurring  in  practice. 

We  support  this  hypothesis  with  evidence  from  case  studies  on  several  Java  APIs  in  wide  practical  use 
(Section  7.1).  We  identify  a  number  of  challenging  recurring  patterns,  all  of  which  can  be  sufficiently 
expressed  with  our  approach.  Furthermore,  the  number  of  annotations  needed  for  capturing  protocols  with 
our  approach  is  small,  especially  when  compared  to  informal  documentation,  and  can  be  provided  quickly. 

The  studied  Java  APIs  arc  all  paid  of  the  Java  standard  library  (that  is  included  with  Java  6)  and  in¬ 
clude  Java  Collections,  Java  Database  Connectivity  (JDBC),  Regular  Expressions,  Exceptions,  and  Charac¬ 
ter  Streams.  According  to  Google  searches,  all  of  these  were  in  wide  practical  use  in  2008  (with  Regular 
Expressions  being  least  frequently  used),  and  we  have  no  indication  that  this  situation  would  change. 
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With  roughly  450  methods,  JDBC  was  the  most  complex  API  we  studied.  We  specified  protocols  for 
JDBC  with  about  2  annotations  per  API  method.  Each  of  these  methods  are  documented  with  approximately 
20  lines  of  English  text  in  the  Java  standard  library.  The  author  could  provide  the  protocol-related  annota¬ 
tions  in  a  short  amount  of  time,  specifying  around  100  methods  per  day,  which  was  largely  spent  reading 
the  extensive  documentation  included  with  the  JDBC  API. 

We  found  three  patterns  to  be  recurring  in  at  least  3  of  the  5  APIs  we  studied:  dynamic  state  tests, 
dependent  objects,  and  method  cases.  These  have  limited  support  in  existing  protocol  checking  approaches 
but  could  be  handled  by  our  approach  in  the  examples  we  studied  (Section  7.4). 

9.1.2  Sound  Modular  Checking 

Our  second  hypothesis  characterizes  the  theoretical  properties  of  our  approach. 

Typestate -based  protocols  with  access  permissions  can  be  verified  in  a  sound  modular  fashion. 

This  hypothesis  is  supported  by  a  formalization  of  our  approach  as  a  type  system  (Chapter  4)  and  a  proof 
of  soundness  (Bierhoff  and  Aldrich,  2007a).  The  proof  of  soundness,  informally  speaking,  guarantees  that 
programs  that  typecheck  in  our  type  system  will  never  violate  declared  protocols  at  run-time. 

The  formalization  demonstrates  a  number  of  properties  that  we  consider  important  for  its  practicality: 

•  Soundness,  i.e.,  a  guarantee  that  the  approach  does  not  miss  any  potential  protocol  violations,  estab¬ 
lishes  the  approach's  well-foundedness. 

•  Modularity,  i.e.,  the  ability  to  check  parts  of  a  program  separately  from  the  rest,  is  crucial  for  scal¬ 
ing  to  large  codebases,  compositional  reasoning  about  programs  developed  by  many  developers,  and 
reasoning  about  libraries  (or  application  “layers”)  separately  from  their  clients. 

•  Refinement  typing  means  that  we  can  check  protocols  on  top  of  a  conventional  Java-like  type  system, 
without  changing  the  execution  semantics  of  the  underlying  language. 

•  Aliasing  flexibility,  i.e.,  the  ability  to  reason  about  the  protocol  of  objects  that  arc  referenced  from 
multiple  places  in  the  program  (modularly).  This  is  the  most  visible  distinguishing  feature  of  our 
approach  and  provides  us  with  crucial  flexibility  for  handling  aliasing  patterns  we  found  inherent  in 
APIs  and  in  the  open-source  software  we  studied. 

•  Support  for  dynamic  tests  allows  developers  to  deal  with  uncertainties  inherent  in  many  APIs  and 
allows  overcoming  imprecisions  in  our  type  system  with  conventional  if -tests. 

•  Flexible  overriding  allows  overriding  methods  without  calling  the  overridden  method,  which  is  com¬ 
monly  done  in  practice. 

These  properties  characterize  our  approach  as  potentially  useful;  the  remaining  hypotheses  establish  its 
applicability  in  practice. 
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9.1.3  Automation 

We  were  concerned  that  our  approach  could  only  be  used  for  manually  proving  programs  correct  with 
respect  to  protocols  (with  hand-written  proofs  such  as  the  ones  shown  in  Chapter  3).  Our  third  hypothesis 
emphasizes  that  our  approach  is  amenable  to  automated  reasoning  with  little  developer  intervention. 

Access  permission-based  protocol  checking  can  be  automated  with  annotation  burden  compa¬ 
rable  to  conventional  type  declarations  and  used  with  conventional  programming  languages. 

We  describe  a  type  inference  system  inspired  by  constraint  logic  programming  that  reduces  permission 
tracking  in  the  type  system  to  linear  constraints  (Chapter  5).  These  constraints  can  be  checked  automatically 
for  satisfiability. 

We  embedded  this  inference  algorithm  into  a  tool.  Plural,  for  checking  protocols  in  the  Java  program¬ 
ming  language  (Chapter  6).  The  tool  requires  developer  annotations  for  method  parameters  and  fields  but 
processes  individual  Java  methods  fully  automatically. 

Measurements  show  that  developer-provided  annotations  arc  in  their  extent  comparable  to  conventional 
Java  typing  information  (both  for  describing  API  protocols  and  for  tracking  objects  from  APIs  in  client 
code).  In  our  case  studies  we  used  about  1  annotation  per  method  of  API  client  code  and  2  annotations 
per  method  for  specifying  the  API  itself.  These  annotations  often  have  the  flavor  of  conventional  types  in 
that  they  “borrow”  a  permission  for  the  method  receiver  or  a  method  parameter.  Borrowing  means  that  the 
permission  that  is  required  by  a  method  is  returned  to  the  caller  when  the  method  returns.  In  other  cases, 
permissions  arc  “consumed”  by  methods,  i.e.,  required  but  not  returned.  These  arc  very  simple  patterns,  and 
our  annotations  can  express  them  efficiently,  making  them  visually  about  as  “large”  as  the  conventional  Java 
types  that  have  to  be  provided  for  every  method  parameter  in  Java. 

Measurements  also  show  that  the  prototype  tool  can  check  protocols  fast  enough  for  interactive  use.  On 
average,  the  tool  checks  individual  methods  in  around  100ms;  the  biggest  method  we  encountered  in  our 
case  studies  was  over  500  lines  long. 

9.1.4  Practical  Checking 

Our  final  hypothesis  claims  applicability  of  our  approach  to  software  found  in  practice. 

The  approach  can  enforce  API  protocols  in  off-the-shelf  object-oriented  software. 

We  used  our  tool  for  checking  API  protocols  in  two  open-source  codebases,  namely  Apache  Beehive 
(Section  7.2)  and  PMD  (Section  7.3.2).  Both  arc  third-party  codebases  that  extensively  use  Java  APIs  we 
specified,  in  particular  JDBC  (Beehive)  and  Java  Collections  (PMD). 

2,000  lines  of  Beehive  code  could  be  checked  by  our  tool  with  5  false  positives  and  while  finding  2 
potential  problems  in  the  code  (that  do  not,  to  our  knowledge,  manifest  in  runtime  errors).  All  but  one  call 
into  JDBC  (due  to  an  invariant  we  could  not  express)  could  be  checked.  Three  usages  of  a  Java  Collection 
through  a  static  field  and  one  reflection-related  warning  account  for  the  remaining  false  positives.  Beehive 
is  itself  intended  to  be  used  as  a  library  in  a  larger  application,  and  we  could  use  our  tool  to  define  a  protocol 
for  using  Beehive  and  establishing  a  correspondence  between  Beehive’s  own  protocol  and  the  protocols  of 
APIs  that  the  Beehive  code  uses. 
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In  PMD,  a  program  of  close  to  40  KLOC,  we  could  successfully  rule  out  98%  of  potential  violations  of 
the  Java  Iterator  protocol  (which  is  part  of  the  Collections  API).  This  required  only  15  annotations,  which 
could  be  provided  by  the  author  in  75  minutes. 

We  also  checked  8.7  KLOC  of  PMD  for  concurrent  modifications  of  collections  while  collections  arc 
iterated,  using  annotations  to  make  assumptions  about  the  rest  of  the  program.  The  close  to  600  annotations 
required  could  be  provided  in  18  hours.  46  false  positives  remain  in  this  part  of  PMD,  which  arc  discussed  in 
detail  (Section  7.3.2).  Many  of  these  false  positives  could  be  avoided;  the  majority  arises  from  an  inheritance 
pattern  that  our  type  system  does  not  support,  but  we  believe  that  our  approach  could  be  modified  to  include 
this  pattern. 

PMD  has  also  been  the  subject  of  case  studies  with  state-of-the-art  whole-program  protocol  analyses 
based  on  tracematches  (Naeem  and  Lhotak,  2008;  Bodden  et  ah,  2008).  Our  approach  successfully  rules  out 
all  false  positives  reported  by  Bodden  et  al.  (2008)  related  to  concurrent  modification  of  iterated  collections 
in  the  paid  of  PMD  we  considered  (Table  7.4).  Precision  of  the  two  approaches  for  checking  compliance  to 
the  simpler  iterator  protocol  is  comparable.  This  suggests  that  “shallow”  protocols  can  be  checked  relatively 
easily  with  both  approaches;  “deep”  protocols  can  cause  problems  for  whole -program  tracematch  analyses 
and  can  be  checked  more  precisely  with  our  approach  at  the  price  of  having  to  annotate  the  program. 

In  certain  cases,  refactorings  enable  checking  protocols  automatically  in  code  that  originally  could  not 
be  analyzed.  Most  of  these  refactorings  arc  to  avoid  shortcomings  in  our  tool  implementation  such  as  a 
relatively  simplistic  local  alias  analysis. 

9.1.5  Thesis 

The  thesis  of  this  dissertation  is: 

Typestate-based  protocols  with  state  refinement  and  access  permissions  can  be  used  for  auto¬ 
mated,  static,  modular  enforcement  of  API  protocols  in  practical  object-oriented  software. 

This  dissertation  has  provided  evidence  for  our  approach’s  well-foundedness  and  its  applicability  to 
common  software  practices,  in  particular  with  respect  to  APIs,  code,  and  how  code  is  written.  We  found 
numerous  opportunities  for  improving  our  tooling,  and  motivation  to  improve  our  approach’s  handling  of 
inheritance.  On  the  other  hand,  our  approach  of  using  access  permissions  to  reason  about  typestate  in  the 
presence  of  aliasing  has  largely  held  up  to  practical  demands.  In  particular,  the  improved  aliasing  flexibility 
has  allowed  us  to  capture  and  enforce  API  protocols  that  could  not  be  captured  with  previous  modular 
program  verification  approaches.  One  of  our  case  studies  suggests  that  our  approach  can  check  protocol 
compliance  in  many  situations  that  cause  imprecisions  in  state-of-the-art  whole-program  protocol  analyses 
(Naeem  and  Lhotak,  2008;  Bodden  et  ah,  2008). 


9.2  Concerns 

Despite  the  evidence  provided  in  this  dissertation,  a  number  of  concerns  remain  regarding  the  potential 
success  of  this  research.  This  section  discusses  the  following  concerns: 

•  Differences  between  programs,  programmers,  and  programming  languages. 
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•  Applicability  to  frameworks. 

•  Expression  cost. 

•  Adoptability. 

•  Tool  performance. 

•  Concurrency. 

9.2.1  Variations  in  Development  Practices 

One  possible  objection  to  this  work  could  be  that  it  may  not  be  applicable  to  all  programs,  all  programmers, 
or  all  programming  languages. 

•  Program  variability.  The  variations  between  programs  can  be  large,  and  it  is  impossible  for  us  to  enu¬ 
merate  all  possible  programs  and  evaluate  how  our  approach  might  work  on  them.  But  our  case  stud¬ 
ies  did  achieve  considerable  breadth  in  covering  common  APIs  and  their  protocol  patterns,  suggesting 
limited  variation  there.  This  suggests  that  our  approach  can  express  many  kinds  of  API  interaction 
patterns  occurring  in  practical  programs. 

•  Programmer  differences.  Programmers  have  different  approaches  to  how  they  solve  problems.  We 
expect  all  programmers  new  to  a  particular  API  to  benefit  from  our  approach.  Experienced  program¬ 
mers  seeking  confirmation  that  their  work  is  correct  will  likewise  benefit  from  our  approach  directly. 
Experienced  programmers  interested  in  the  quickest  possible  solution  may  benefit  from  our  approach 
by  saving  testing  time  and  pointing  out  problems  on  rarely  exercised  code  paths.  Less  desirably,  de¬ 
velopers  may  choose  to  ignore  protocol  violation  warnings,  unlike  conventional  typing  errors,  since 
they  do  not  prevent  the  program  from  being  run. 

•  Programming  language  dependencies.  The  approach  presented  in  this  dissertation  was  designed  for 
conventional  object-oriented  languages.  While  our  tool  was  built  specifically  for  Java,  we  see  little 
problem  in  using  the  approach  with  C#  or  C++  (with  additional  support  for  function  pointers).  Imper¬ 
ative  languages  without  subtyping  (such  as  C)  could  also  benefit  from  this  approach.  The  soundness 
of  our  approach  depends  on  type  safety,  but  it  will  work  with  weakly  typed  languages  such  as  C  and 
C++.  Dynamically  typed  languages  (such  as  SmallTalk,  Python  or  JavaScript)  would  require  more 
significant  adaptation. 

Most  functional  languages  support  shared  mutable  state,  but  common  belief  is  that  it  is  rarely  used. 
However,  functional  programs  interact  with  the  world  using  elementary  libraries  that  give  for  instance 
access  to  files,  the  screen,  or  databases.  These  interactions  follow  similar  protocols  to  the  ones  de¬ 
fined  in  the  Java  standard  library,  and  our  approach  can  be  used  to  enforce  them.  Furthermore,  our 
case  studies  show  that  protocols  of  elementary  APIs  arc  not  limited  to  the  fringes  of  programs  but 
affect  higher-level  code  as  well,  suggesting  that  our  approach  could  significantly  help  functional  pro¬ 
grammers  in  following  protocols. 
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9.2.2  Applicability  to  Frameworks 

Software  frameworks  have  made  significant  inroads  in  the  development  of  GUI-based  desktop  and  Web 
applications,  among  other  domains.  Frameworks  provide  an  application  skeleton  to  which  application  de¬ 
velopers  can  add  “plug-ins”  (extension  code)  that  interacts  with  the  framework  code  to  enhance  or  change 
the  framework’s  default  functionality.  Frameworks  typically  require  plug-ins  to  implement  certain  “lifecycle 
methods”  that  the  framework  invokes  at  well-defined  moments.  From  a  plug-in’s  perspective,  a  framework 
defines — often  overwhelmingly  many — APIs  that  the  plug-in  can  use  to  call  back  into  the  framework  during 
the  execution  of  a  plug-in  lifecycle  method  (Fairbanks,  2007);  plug-ins  act  as  API  clients  in  this  regal'd.  In 
order  to  allow  this  interplay,  frameworks  and  plug-ins  share  objects  (through  which  lifecycle  methods  can 
be  invoked  and  callbacks  into  the  frameworks  can  be  performed). 

The  protocols  for  framework-plug-in  interactions  are  often  intricate,  and  it  appeal's  that  many  of  them 
could  be  formalized  and  enforced  using  typestates  (Jaspan  and  Aldrich,  2009).  In  this  sense  our  approach 
therefore  complements  recent  research  on  helping  developers  accomplish  their  goals  with  the  “right”  frame¬ 
work  interactions,  e.g.,  using  “design  fragments”  (Fairbanks,  2007):  our  approach  allows  checking  whether 
the  concrete  combination  of  framework  interactions  chosen  by  a  developer  is  permitted  by  the  framework. 
Furthermore,  this  may  raise  the  awareness  of  framework  developers  for  the  protocols  they  impose,  leading 
to  potentially  cleaner  framework  designs  that  what  we  see  in  practice  today  (Fairbanks,  2007). 

9.2.3  Expression  Cost 

Our  case  studies  show  that  annotations  for  using  our  tool  have  the  same  extent  as  conventional  typing 
information  when  tracking  every  object  in  a  program.  This  is  a  significant  cost  that,  as  our  case  studies 
show,  can  be  hard  to  retro-fit  into  existing  code.  The  case  studies  also  indicate  that  our  approach  can  be 
incrementally  phased  into  existing  code,  by  starting  with  one  simple  API  protocol  in  a  part  of  the  codebase 
and  expanding  out  from  there.  Given  the  similarity  to  conventional  typing,  it  appeal's  that  annotations  can 
be  realistically  provided  from  the  start  in  newly  written  software,  but  this  intuition  should  be  confirmed  in 
future  research. 

9.2.4  Adoptability 

The  approach  presented  in  this  dissertation  proposes  of  a  number  of  non-trivial  concepts,  including  5  dif¬ 
ferent  kinds  of  permissions,  state  refinement,  state  guarantees,  a  distinction  between  “frame”  and  “virtual” 
permissions,  and  the  use  of  “state  invariants”.  Developers  cannot  be  assumed  to  be  familial'  with  these 
concepts,  creating  challenges  for  adopting  this  work  into  practice. 

Many  of  our  concepts  are  designed  to  mirror  intuitive  notions  used  by  developers  today.  For  instance,  we 
have  evidence  from  the  Java  Streams  library  that  the  developers  of  this  library  documented  state  invariants 
informally  (Bierhoff  and  Aldrich,  2005).  Additionally,  the  concept  of  state  refinement  is  already  familiar  to 
many  developers  from  Statecharts  as  they  are  part  of  the  UML  (Rumbaugh  et  al.,  2004).  But  the  question 
remains  how  easy  it  is  to  understand  these  concepts  in  the  first  place. 

We  believe  that  the  biggest  challenge  in  explaining  this  work  to  others  is  the  systematic  distinction 
between  read-only  and  read-write  access  (which  is  pervasive  in  the  approach),  followed  by  the  distinction 
between  frame  and  virtual  permissions  (which  is  relevant  for  subclassing).  While  treating  read-only  access 
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special  is  standard  in  program  verification  (Barnett  et  al.,  2004)  and  functional  programming,  it  appears  to 
be  largely  implicit  in  conventional  imperative  and  object-oriented  programming  practices. 

We  could  retrofit  permissions  quite  well  into  existing  object-oriented  programs.  Moreover,  collaborators 
at  Carnegie  Mellon  University  have  adopted  our  approach  for  their  own  research  (Beckman  et  al.,  2008,  and 
others),  and  the  approach  has  been  used  in  the  classroom  and  in  the  AcmeStudio  project  at  Carnegie  Mellon. 
This  suggests  that  the  approach  is  compatible  with  implicit  programming  notions  and  accessible  at  this 
point  to  well-qualified  developers.  Part  of  these  successes,  we  believe,  is  due  to  our  efforts  to  largely  shield 
developers  from  the  complexities  of  using  linear  logic  directly.  In  the  future,  we  hope  to  further  simplify  our 
model  for  application-specific  purposes.  We  also  plan  to  investigate  adoptability  challenges  more  thoroughly 
in  user  and  field  studies. 

9.2.5  Tool  Performance 

We  noticed  that  subjective  delays  in  running  the  tool  can  occur  in  two  situations:  extensive  unpacking  and 
packing  due  to  field  accesses  interspersed  with  method  calls,  and  extensive  use  of  method  cases.  Both 
situations  permit  many  possible  choices  (such  as,  which  state  to  pack  to,  or  which  method  case  to  choose). 
If  these  choices  are  not  determined  by  the  program  then  the  tool  has  to  carry  them  forward,  which  can  lead 
to  snowball  effects  of  choices  multiplying. 

We  do  not  think  that  these  issues  are  a  threat  to  the  approach’s  practicality:  we  see  them  only  in  rare 
cases,  they  only  occur  in  severely  underspecified  programs,  and  they  arc  partially  a  result  of  our  tool  not 
being  optimized  for  these  situations.  Furthermore,  these  are  problems  with  the  tool  that  arc  largely  due  to 
our  desire  to  analyze  methods  fully  automatically,  simple  programmer-provided  information  could  eliminate 
these  problems  by  telling  the  tool  which  choice  to  make  (which  state  to  pack  to,  or  which  method  case  to 
choose).  In  fact,  we  believe  that  such  annotations  would  capture  valuable  programmer  intent  and  could 
easily  become  part  of  future  tools  or  programming  languages  based  on  our  approach. 

Our  tool  checks  constraint  satisfiability  based  on  Fourier-Motzkin  elimination,  which  creates  an  expo¬ 
nential  number  of  formulas  in  the  worst  case  when  eliminating  variables.  We  experience  good  performance 
in  practice,  which  seems  to  result  from  a  carefully  chosen  order  of  eliminating  variables  (Section  5.2.2). 

The  alternating  quantification  in  constraint  formulas  prevents  us  from  using  linear  programming  for 
checking  constraint  satisfiability,  as  was  done  in  previous  work  (Terauchi,  2008).  Alternating  quantifica¬ 
tion  is  a  well-known  challenge  for  SMT  (Satisfiability  Modulo  Theories)  solvers.  Other  researchers  have 
successfully  used  Farkas’  lemma  to  remove  alternating  quantification  (Gulwani  and  Tiwari,  2008;  Gulwani 
et  al.,  2008).  This  would  enable  the  use  of  linear  programming  tools  at  the  price  of  being  incomplete  (Gul¬ 
wani  et  al.,  2008). 

9.2.6  Concurrency 

This  work  does  not  enforce  correct  synchronization  when  accessing  shared  data  in  concurrent  programs: 
our  analysis  assumes  single-threaded  execution  or  at  least  a  correctly  synchronized  multi-threaded  program. 
Nels  Beckman  has  proven  a  variant  of  our  approach  sound  for  concurrent  programs  with  Atomic  blocks 
(Beckman  et  al.,  2008)  and  extended  the  Plural  tool  to  such  programs  in  his  tool,  NIMBY1 .  A  key  insight  of 
his  work  is  to  use  aliasing  (as  encoded  with  permissions)  as  a  sound  approximation  of  thread  sharing,  which 

'NIMBY  is  available  together  with  Plural  at  http:  //code  .google .  com/p/pluralism/. 
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forces  the  correct  placement  of  Atomic  blocks  where  thread-shared  data  is  accessed  or  interleavings  with 
other  threads  would  invalidate  state  information  assumed  by  the  current  thread. 

9.3  Discussion 

We  made  a  number  of  observations  while  investigating  our  approach  in  this  dissertation  that  we  discuss  in 
this  section,  including  the  possible  pay-off  from  using  our  approach,  its  most  important  features  in  practice, 
advice  we  can  give  to  API  designers,  and  the  effects  we  believe  this  work  can  have  on  software  engineering 
practice. 

9.3.1  Pay-off  and  Incrementality 

Since  many  of  the  problems  our  approach  catches  will  be  found  during  testing,  the  added  benefit  of  applying 
our  approach  after  the  fact  to  deployed  code  is  small  unless  it  is  vital  that  the  code  being  checked  is  correct. 

Much  greater  benefit  comes  from  using  the  approach  during  development  and  maintenance  tasks,  which 
our  modular  approach  permits  easily.  There  arc  several  situations  in  which  we  expect  the  benefit  to  be 
particularly  large: 

•  Gaining  familiarity  with  a  API.  Rather  than  having  to  extensively  read  API  documentation  (if  even 
available)  developers  trying  to  use  an  API  they  arc  not  familial-  with  will  be  able  to  “just  start  using” 
the  API,  and  our  approach  will  tell  developers  when  they  violate  protocols  associated  with  the  API. 
For  complex  APIs  this  strategy  can  be  significantly  faster,  in  particular  when  important  protocol  rules 
are  documented  in  unexpected  places. 

•  Maintenance  and  agility.  Any  code  change,  especially  by  a  developer  that  did  not  originally  write  the 
code  being  changed,  may  violate  existing  protocols.  Our  approach  avoids  the  overhead  of  extensively 
re-testing  the  program  after  every  code  change.  This  also  makes  our  approach  attractive  for  use  in  agile 
software  development,  despite  the  extra  time  it  may  take  developers  to  provide  needed  annotations. 

•  Intermediary  library  implementation.  Our  approach  seems  to  lend  itself  to  establishing  consistency 
between  a  library’s  protocol  and  the  protocols  of  APIs  used  by  the  library,  as  we  saw  with  Beehive. 

•  Complexity.  Obviously,  it  is  harder  to  follow  complex  protocols,  in  particular  those  involving  multiple 
objects,  than  straightforward  protocols.  Our  approach  seems  to  be  able  to  capture  and  enforce  such 
protocols. 

Additionally,  our  approach  seems  to  support  graceful  ways  into  using  it:  checking  protocols  of  different 
APIs  in  a  given  codebase  is  largely  orthogonal  (as  we  saw  with  Beehive),  simpler  protocols  are  more  easily 
checked  (as  we  saw  with  PMD),  and  protocols  can  be  checked  in  only  a  part  of  the  code  (as  we  also  saw 
with  PMD). 

9.3.2  Essential  Features 

While  we  designed  our  approach  to  be  as  flexible  as  possible,  there  are  certain  features  that  we  found 
essential  for  modeling  and  enforcing  protocols  in  practice: 
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•  Borrowing  and  capturing  objects  (Section  6.4.2).  Methods  (and  constructors)  seem  to  either  borrow 
or  capture  permissions  to  parameters.  This  means  that  methods  return  either  the  same  permission 
that  they  require,  or  no  permission  at  all,  although  other  situations  arc  certainly  conceivable.  When 
permissions  arc  captured  they  often  end  up  in  invariants  of  another  object,  which  we  discuss  below. 
Plural  provides  annotations  like  Full  that  capture  these  two  cases  conveniently. 

•  State  dimensions  and  state  refinement  (Section  3.1).  We  found  our  hierarchical  notion  of  state  ma¬ 
chines  convenient  for  modeling  protocols,  and  they  seem  to  make  specifications  more  concise,  which 
we  believe  is  important  for  adoptability  in  practice. 

•  Support  for  dynamic  state  tests  (Section  6.3.6).  Dynamic  state  tests  arc  ubiquitously  used  in  API 
protocols,  and  support  for  them  is  essential  for  ensuring  correct  protocol  usage. 

•  State  guarantees  (Section  3.2)  and  object  dependencies  in  general.  API  protocols  in  practice  seem  to 
go  beyond  simple  state  machines  for  individual  objects  and  include  multiple  interrelated  objects.  Per¬ 
missions  let  us  model  these  dependencies  and  we  think  that  this  flexibility  will  be  crucial  in  modeling 
many  API  protocols,  even  in  the  context  of  more  complete  behavioral  models  such  as  first-order  logic. 
The  idea  of  state  guarantees  was  crucial  for  expressing  dependencies  in  JDBC  in  particular,  and  it  will 
be  interesting  whether  state  dependencies  can  be  generalized  beyond  finite  state  machines. 

•  Capturing  permissions  for  the  lifetime  of  another  object  or  reference  (Section  7.1.2).  Capturing  ob¬ 
jects  for  the  lifetime  of  another  object  let  us  express  dependencies  between  objects  in  APIs  (see 
above).  In  another  sense,  this  can  be  seen  as  lending  an  object  to  another  object  (instead  of  a  method, 
as  above).  We  can  also  borrow  permissions  “from”  another  object  (Boyland  et  al.,  2007)  for  the  life¬ 
time  of  the  reference  holding  the  borrowed  object,  by  leaving  the  borrowing  object  unpacked  for  that 
time. 

•  Local  views  on  objects.  Permissions  allow  us  to  talk  about  what  we  know  about  an  object  from  the 
perspective  of  a  given  method.  They  encode  what  other  references  may  do  without  explicit  knowledge 
of  where  these  references  may  be.  This  facilitates  compositional  automated  reasoning.  It  also  seems 
to  line  up  well  with  a  programmer’s  way  of  thinking  about  a  given  reference  from  the  perspective  of 
the  code  module  she  works  on. 

•  Distinguishing  read-only  access  through  a  particular  reference.  This  feature  of  permissions  represents 
important  design  intent  that,  for  instance,  allows  us  to  talk  about  whether  or  not  an  iterator's  hasNext 
implementation  is  allowed  to  modify  the  iterator  (Section  7.2.2).  It  also  permits  “strong  updates” 
in  the  presence  of  aliasing  (with  full  permissions),  which  is  an  improvement  over  lineality -based 
approaches. 

9.3.3  Advice  to  API  Designers 

From  our  case  studies  on  Java  APIs  we  extract  a  number  of  lessons  for  API  designers  based  on  patterns  that 
were  hard  or  awkward  to  express  with  our  approach. 

•  Avoid  methods  with  multiple  purposes.  Important  design  intent  is  lost  when  methods  implicitly  have 
multiple  outcomes.  For  instance,  there  is  only  one  method  for  creating  an  iterator  over  a  collection. 
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and  it  is  not  clear  whether  this  iterator  will  modify  the  iterated  collection  or  not  (Bierhoff,  2006). 
While  method  cases  can  elegantly  handle  such  methods,  we  believe  their  behavior  is  not  easily  pre¬ 
dictable  from  code  that  uses  them. 

•  Document  “mode"  restrictions  on  methods  with  these  methods.  In  our  work  with  JDBC’s  ResultSet 
interface  we  discovered  several  dozen  methods  into  the  case  study  that  there  were  two  “modes”  to  be 
distinguished  that  affected  previously  specified  methods  (Section  7.1.1).  However,  the  affected  meth¬ 
ods’  documentation  did  not  indicate  this  mode  dependency.  Such  restrictions  should  be  documented 
with  all  methods  that  arc  affected. 

Furthermore,  if  modes  exist  then  methods  should  not  just  be  marked  as  “possibly  unavailable”.  In  the 
Java  Collections  API,  many  methods  arc  marked  as  “optional”  without  specifying  exactly  under  which 
circumstances  the  methods  will  work.  It  turns  out  that  these  methods  arc  available  for  “modifiable” 
collections  (Section  7.1.2),  and  we  believe  it  would  be  helpful  to  state  this  in  the  documentation. 

•  Avoid  subtypes  that  do  not  fully  follow  inherited  protocols.  In  the  Java  Streams  API  it  appeal's  that 
“pipes”  have  a  more  restrictive  protocol  that  any  old  stream:  while  other  streams  can  be  closed  at  any 
time,  sinks  (instances  of  PipedlnputStream)  can  only  be  closed  when  the  end  of  the  stream  was 
detected  (Section  3.2),  violating  behavioral  subtyping  (Liskov  and  Wing,  1994).  This  problem  can  be 
avoided  with  a  more  restrictive  specification  of  the  basic  stream  protocol  and  being  more  permissive 
in  subtypes  such  as  FilelnputStream  that  can  definitely  be  closed  at  any  time.  But  problems  re¬ 
main  when  such  streams  are,  for  instance,  wrapped  into  a  Buff eredlnputStream.  We  believe  that 
behavioral  subtyping  should  be  respected  in  APIs  because  using  abstract  supertypes  defined  in  APIs 
can  otherwise  lead  to  runtime  errors  that  are  hard  to  understand. 

9.3.4  Effect  on  Software  Engineering  Practice 

We  hope  that  this  work  will  have  the  following  effects  when  adopted  into  software  engineering  practice: 

•  Cleaner  APIs.  Specifying  permission-based  protocols  for  APIs  should  make  these  APIs  cleaner  be¬ 
cause  awkward  protocols  or  aliasing  will  become  apparent. 

•  Confident  aliasing.  Developers  will  be  able  to  more  confidently  use  aliasing  in  their  programs  be¬ 
cause  an  automated  tool  helps  them  predict  the  non-local  effects  of  aliasing.  We  believe  that  this  has 
significant  potential  for  easing  software  construction. 

•  Rapid  API  use.  Tool  support  will  allow  rapid  and  correct  use  of  unfamiliar  APIs,  facilitating  collab¬ 
oration  (defined  broadly).  This  could  increase  the  number  of  APIs  being  used  in  a  project,  or  simply 
accelerate  software  construction. 

•  Facilitation  of  reuse.  This  work  makes  it  easier  to  make  a  code  module  reusable  as  an  API.  It  will  be 
easier  to  document  how  to  use  the  module.  Protection  against  protocol  violations  is  either  unnecessary 
(in  trusted  codebases)  or  could  potentially  be  automatically  generated. 

•  Dependability.  Static  assurance  of  protocol  compliance  will  lead  to  increased  dependability  both  of 
the  client  side  and  the  provider  side  of  APIs. 
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9.4  Contributions 

Novel  abstractions:  State  refinement  and  access  permissions.  New  abstractions  are  proposed  for  pro¬ 
tocol  specification  and  enforcement  based  on  typestates  that 

•  improve  expressiveness,  handling  of  subtyping  and  inheritance,  and  reasoning  under  aliasing  com¬ 
pared  to  previous  typestate-based  approaches. 

•  can  be  used  with  existing  object-oriented  programming  languages. 

These  abstractions  allow  more  succinct  protocol  specifications  (by  avoiding  state  explosion  problems 
and  conveniently  encoding  uncertainty)  and  support  important  API  protocol  patterns  that  arc  insufficiently 
supported  in  previous  protocol  checking  approaches. 


Sound  modular  protocol  checking  under  aliasing.  Protocol  checking  based  on  access  permissions  is 
formalized  as  a  modular  type  system.  A  fragment  of  this  system  was  proven  sound  (Bierhoff  and  Aldrich, 
2007a).  Due  to  the  use  of  access  permissions,  protocol  compliance  can  be  checked  in  a  modular  fashion 
even  if  objects  can  be  manipulated  though  aliases  from  elsewhere  in  the  program. 


Inference.  An  inference  system  for  syntax-directed  tracking  of  polymorphic  access  permissions  is  pre¬ 
sented.  The  inference  system  combines  resource  tracking  techniques  from  linear  logic  programming  with 
constraint  logic  programming  to  reduce  protocol  checking  to  quantified  linear  constraints,  which  can  be 
checked  for  satisfiability  using  Fourier-Motzkin  elimination. 


Practical  tooling.  A  working  prototype  tool  demonstrates  that  the  approach  can  be  used  in  practical  au¬ 
tomated  software  development  tools  for  conventional  programming  languages.  Performance  measures  in¬ 
dicate  the  tool  to  be  suitable  for  interactive  use.  A  number  of  extensions  to  the  proposed  approach  were 
implemented  in  the  tool  that  make  it  more  useful  in  practice. 


Evaluation.  Case  studies  evaluate  the  approach’s  ability  to  capture  and  enforce  commonly  used  API  pro¬ 
tocols  in  third-party  open-source  software.  Developer-provided  design  intent  is  in  its  extent  comparable  to 
conventional  types,  and  recurring  API  patterns  can  be  successfully  captured. 


9.5  Future  Work 

There  arc  many  avenues  for  future  research  directly  related  to  this  work. 

First,  one  could  apply  this  work  specifically  to  a  number  of  well-known  problems  including  the  follow¬ 
ing,  all  of  which  have  received  significant  attention  on  their  own: 

•  Object  initialization  protocols  (Fahndrich  and  Xia,  2007;  Qi  and  Myers,  2009)  arc  a  special  case  of 
protocols. 
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•  Static  avoidance  of  null  dereference  errors  (Fahndrich  and  Xia,  2007)  and  buffer  overruns  (Hackett 
et  al.,  2006)  may  be  increased  in  cases  where  pointer  properties  are  state-dependent.  For  instance, 
fields  may  have  to  be  non-null  in  certain  states,  which  we  can  encode  with  Plural. 

•  Memory  corruption  (through  pre-mature  memory  deallocation)  and  memory  leaks  (through  missing 
memory  deallocation)  in  programming  languages  with  explicit  memory  deallocation  can  be  avoided 
by  ensuring  that  a  unique  permission  is  available  for  deallocating  memory  (which  prevents  corrup¬ 
tion),  and  that  unique  permissions  are  not  silently  forgotten  (which  prevents  leakage  in  many  cases). 

•  Information  flow  tracking  could  be  encoded  with  “marker  states”.  For  instance,  we  may  associate  data 
with  a  “secret”  marker  state  (type  qualifier)  and  make  sure  that  secret  or  possibly-secret  data  does  not 
become  public. 

•  SQL  injection  could  be  statically  prevented  with  the  following  classification  of  user  data  as  “safe”  or 
“harmful”,  which  accounts  for  dynamic  checking  that  user  data  is  in  fact  harmless: 

@States(value  =  {"safe",  "harmful"},  dim  =  "dangerous",  marker  =  true) 

public  interface  UserData  { 

@Pure("dangerous") 

@TrueIndicates("safe") 
boolean  isSafe(); 

@Imm 

@ResultUnique(ensures  =  "safe")  UserData  makeSafet); 

} 


•  Implementations  of  security  protocols  could  be  checked  for  compliance  with  the  desired  protocol. 
This  is  in  particular  useful  for  avoiding  easy-to-miss  loopholes  such  as  omitted  runtime  checks  in 
“secure”  clients.  For  instance,  the  Kerberos  protocol  requires  clients  to  generate  fresh  tokens  that  are 
later  compared  to  answers  from  the  Kerberos  server  (Neuman  and  Ts’o,  1994).  If  these  tokens  are 
not  in  fact  fresh,  or  are  not  compared  later,  then  the  client  becomes  vulnerable  to  replay  attacks.  This 
work  can  be  used  to  enforce  the  comparison,  and  it  can  put  restrictions  on  what  data  is  considered 
fresh  (for  instance,  constants  could  be  forbidden). 

Second,  we  hope  to  develop  techniques  for  further  reducing  the  annotation  burden  of  using  Plural. 
One  promising  way  of  doing  so  could  be  to  turn  our  dataflow  analysis  for  tracking  permissions  into  an 
interprocedural  analysis  which,  for  instance,  could  check  an  entire  compilation  unit  or  larger  code  module 
fully  automatically. 

Third,  our  approach  promises  to  allow  performance  improvements:  it  can  detect  unnecessary  runtime 
tests  in  programs,  which  could  be  automatically  removed  by  a  compiler.  This  may  in  particular  simplify  li¬ 
brary  implementations,  which  will  need  to  perform  fewer  runtime  checks  to  ensure  that  clients  are  using  the 
library  according  to  its  protocols.  Preliminary  measurements  show  that  iterators  over  j  ava  .util.  Arr  ayLi  st , 
an  array-based  list  implementation  in  the  Java  Collections  API,  speed  up  between  14%  and  20%  (depending 
on  the  length  of  the  list,  p  <  0.01  over  20  runs)  if  they  do  not  perform  defensive  runtime  checks.  On  the 
other  hand,  if  security  is  an  issue,  we  may  want  to  generate  such  runtime  checks  to  prevent  possible  exploits. 
An  interesting  research  question  is  which  checks  in  current  library  implementations  are  needed  for  security 
reasons  and  which  ones  are  not. 


9.6.  SUMMARY 
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Finally,  the  time  has  come  to  perform  user  and  field  studies  to  investigate  the  effects  of  this  work  on 
software  development  and  maintenance.  In  this  context  it  will  become  crucial  to  make  protocol  specifica¬ 
tions  for  APIs  available  to  interested  users.  Protocol  mining  techniques  could  be  used  to  synthesize  typestate 
protocols,  and  subsequent  static  checking  results  could  improve  the  accuracy  of  these  protocols. 

In  addition,  this  research  may  inform  other  lines  of  research. 

First,  this  approach  can  be  used  to  aid  engineers  in  developing  concurrent  software.  Nels  Beckman  is 
successfully  employing  permissions  to  ensure  race  freedom  in  concurrent  programs  that  use  Atomic  blocks 
for  mutual  exclusion  (Beckman  et  ah,  2008).  Even  if  no  protocols  arc  tracked,  permissions  can  still  enforce 
the  consistent  use  of  Atomic  blocks  (or,  potentially,  locks)  when  accessing  shared  mutable  data,  which  we 
think  is  a  very  exciting  and  promising  result. 

Second,  a  very  exciting  avenue  for  future  research  is  the  use  of  permissions  in  program  verification.  The 
JML  and  Spec#,  two  leading  approaches  in  this  space,  use  ownership  to  enable  sound  modular  program 
verification  (Barnett  et  ah,  2004;  Died  and  Muller,  2005).  It  appeal's  that  permissions  can  track  first-order 
logic  predicates  just  like  typestates  (Bornat  et  ah,  2005;  Bierhoff  and  Aldrich,  2008;  Leino  and  Muller, 
2009).  This  would  bring  the  dexibility  that  permissions  provide  for  reasoning  about  dependent  objects  to 
traditional  program  veridcation.  Meanwhile,  our  full  permissions  are  remarkably  similar  to  the  “owner-as- 
modider”  paradigm  used  for  verifying  programs  in  Spec#  and  the  JML,  suggesting  that  permissions  could 
largely  supersede  ownership  in  program  veridcation.  This  in  particular'  since  transferring  permissions  be¬ 
tween  objects  is  simpler  than  transferring  ownership.  This  is  because  permission  transfer  is  a  local  operation 
between  the  provider  and  the  consumer  of  a  permission,  while  ownership  is  a  globally  visible  property  whose 
modidcations  can  consequently  have  global  impact  (Muller  and  Rudich,  2007).  We  believe  that  separation 
guarantees  between  different  pieces  of  owned  data  transfer  to  permissions:  when  data  is  only  modided 
through  full  permissions  then  the  state  information  associated  with  share  permissions  cannot  have  changed. 

Third,  the  techniques  developed  for  tracking  permissions  in  Plural  might  be  useful  to  automate  reasoning 
in  separation  logic.  Fractional  permissions  have  been  combined  with  separation  logic  (Bornat  et  ah,  2005), 
but  separation  logic  is  undecidable  and  often  relies  on  manual  proofs.  Static  analyses  have  been  successful 
in  deriving  separation  logic  constraints  (Magill  et  ah,  2006;  Calcagno  et  ah,  2009),  and  we  believe  that  our 
permission  inference  could  be  used  to  include  fractional  permissions  into  these  approaches. 

Finally,  it  appears  that  typestates  can  be  fruitfully  combined  with  other  areas  of  research.  For  instance, 
software  model  checking  may  scale  to  larger  programs  when  using  typestates  to  abstract  the  behavior  of 
software  components.  Model  checking  may  subsequently  be  used  to  check  higher-level  properties  than 
protocol  compliance  of  API  objects.  Test  case  generation  may  also  benedt:  it  is  currently  very  difdcult 
to  test  higher-level  classes  automatically  that  rely  on  input  data  in  a  particular  “state”.  Typestate -based 
protocols  may  simplify  automatic  test  data  generation  in  these  situations.  Finally,  it  may  be  possible  to 
trace  high-level  business  rules,  such  as  “an  account  cannot  be  overdrawn,”  into  code  using  typestates,  which 
would  establish  a  better  connection  between  requirements  and  code. 


9.6  Summary 

This  dissertation  aims  to  provide  comprehensive  help  to  developers  in  using  and  providing  APIs  with  asso¬ 
ciated  usage  protocols  based  on  typestates  (Strom  and  Yemini,  1986).  The  dissertation  proposes  the  novel 
techniques  of  state  rednement  and  access  permissions  to  overcome  challenges  in  expressiveness,  handling 
of  subtyping  and  inheritance,  and  aliasing  dexibility  in  previous  protocol  checking  approaches.  It  provides  a 
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static  technique  for  sound  and  modular  protocol  checking  in  conventional  object-oriented  software  that  can 
be  automated  in  practical  tools  for  interactive  use. 

The  dissertation  describes  a  prototype  tool.  Plural,  that  plugs  into  the  Eclipse  development  environment 
to  check  protocol  compliance  in  conventional  Java  code.  Plural  can  check  protocols  in  our  case  studies 
with  sufficient  performance  and  precision.  Plural  is  open-source  and  uses  Java  5  annotations  to  capture 
protocol-related  design  intent  directly  in  Java  source  code. 

Case  studies  with  APIs  from  the  Java  standard  library  indicate  sufficient  expressiveness  of  the  presented 
approach  for  practical  protocols.  Case  studies  with  open-source  programs  indicate  high  precision  and  low 
developer  overhead  when  using  the  presented  approach  for  checking  protocol  compliance  in  off-the-shelf 
software.  The  additional  protocol -related  design  intent  of  the  presented  has  the  extent  and  feel  of  conven¬ 
tional  typing  information. 

This  dissertation  shows  that  state  refinement  and  access  permissions  capture  real  API  protocols,  can 
be  used  for  sound  modular  protocol  enforcement,  can  be  automated  in  tools  for  developers,  and  can  check 
protocol  compliance  in  practical  object-oriented  software.  These  results  suggest  that  the  presented  approach 
could  be  used  in  practical  software  development,  and  that  permissions  could  form  the  basis  of  future  type 
systems  for  practical  programming  languages. 
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